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PREFACE 



In 197d, the National Institute of Education embarked upon an ambi- 
tious three year demonstration that was intended to field' test new models of 
how best to provide schools with high quality information and technical 
assistance to help them solve locally ^.dentified problems. This effort, 
called the R&D Utilization (RDU) program, also included a significant re- 
search component, j^^hich operated bajth withm seven f ended service delivery 
projects, and through a three and a half ye^r study, one part of which is 
reported m this volume. The study of the ROD program was not intended as a 
traditional evaluation of the- degree to which demonstration objectives were^ 
met by each of the aeven funded projects.' Rather, the study nad a more 
general mandate to use th| experiences of the ROD projects and the schools 
that became their clients to illurflmate sgme enduring problems jin creating 
and sustainuig effective thanqe programs m schools- which have been voiced by 
researchers^, policy makers, program managers and practitioners. 

This general mandate has led to a wide variety of different reports, 
each of which addresses the general question of how to produce' effective 
knowledge^use and school improvement m schools from a different • perspective, 
or for a different' audience. This volume is intended ^to present an analysis 
of the strategies that were employed m the ROU program, and th^ir impacts at 
the school level* While the question of program impacts is an evaltietive 
one, our objectives must be viewed against a bioader backdrop. "Our approach 
to the- task of "expl ining" ROU and its impacts on schools has been deliber- 
ately eclectic m three different ways: we have attempted to combine ((1) 
policy research with more^ general social research orientations; (2) concep- 
tual and empirical approaches to the phenomenon of managed school change; and 
(3) a research strategy that attempts to meld both quantitative and qualita- 
tive data throughout the analysis and withm each chapter. Because these 
three forms of eclecticism are critical to the work, we would like to elabor-' 
ate a bit more on the meaamg of each for <^he study of the ROU program as it 
affected schools. , ^ ' . 

Both policy and social science research require empirical data to 
support their conclusions but, m general, jthe rules of evidence required for 
the latter are more stringeqt than for the former. /There are sound reasons 
for the distinction: policy research, after all, is intended to feed into 
decision-making processes, and modestly supported conclusions m this context 
ace better than no information. In science, on. the other hand, incremental 
knowledge-building is the norm. In our work, we. have chosen to take both 
large leaps and to make small steps m - incremental knowledge. While, in 
some cases we expect that the more' rigorous social scientist may be ^noyed, 
we do 80 because our work i^ intended to feed into the world of policy and 
practice. If we do not draw the inferences that we believe appropriate, \wiio 
will? On the other hand, we are convinced that" our work is also veil ground- 
ed m social science disciplinary bases, and, m particular, much of our 
evidence is so overwhelmingly strong that careful elaboration is appropriate 
m order to draw attention to its potential for contributing to basic knowl- 
edge . 



Another 'major difference between . policy and social science research 
IS m the degree to which the researcher becomes an advocate for "social 
solutions" with the soical scientist providing "implications for future 
research" and the policy researcher "i^mplications for legislation or program 
improvement.** Again,, we have chosen at some point to draw out the implic^-* 
tions of our "findings for potential practice. At other times, we do not, 
although 'many policy researchers might feel impelled to. In this report we 
have, on the whole, tended .to draw away from policy recommendations, but in 
instances where we believe the data are both strong and relevant, we have* 
made some assertions. We acknowledge this ambivalence, and note only that il 
reflects an inability on our part to draw totally away from pract ical , impli- 
cations of our findings even m what is* largely a research monograph. 

We have attempted to move between conceptual analysis and a detailed ^ 
presentation of empirical data. In doing so, we believe that we may be m a 
position where w& will please neither the theoretician wh« may find our 
.middle-range theories trivial, nor%the committed empiricist who will con- 
sistently cry, **but tell' me what it really looks Mke out there — give me some 
examples'.** One of our reviewers , has, m fact, warned uS of the problems of | 
flying at 10,000 feet: one \^ not high enough to rxse- above the turbulent 
air, but too high to see things clearly. Again, we can only 'state that we. 
have tried to provade enough descriptive information to ground the attempt to 
provide a conceptual framework and an analysis which reveals a range of 
possible causal relationships,^ m the hopes that this will make the study of 
RDU of >Dore lasting significance to both policy makers and. researchers. 

^ In addition, we have put ourselves m the position where there may be 

cdhfusion aa to Cl) where we are introducing empirical data to supports a 
prion conceptual frameworl<s; (2) where they. are used to explain, post 
fact umj ^ findings that were not anticipated, ' and (3) -where we are, as all^ 
researchers, engagm^m **informed speculation,** without the benefit of solid 
empirical evidence. We have tried to indicate throughout the vblume, where ^ 
we are following our conceptual model that was used to formulate'the study, 
where we are elaborating on it based on accretions of knowledge oyer the 
course of the project, .and where we are being most speculative. In particu- 
lar, we have attempted to label our speculations arfd to refer to other 
research rather than simply to our own imaginative powers. 

A final dilemma for our readers is m the way in which we have 
attam^fted to bland qualitative and quahtitatTve data. • Our approach is, by 
most disciplinary standards, unorthodox. We believe our attempts to create 
data from a Variety of different sources, and our attempt to use a range of 
different analyti9 techniques is requir.ed,in order to addr^ess the variety of 
questions that can be asked about a complex 'social pfocess. On the other 
hand, while we have gained through our 'approach of using each bit of data 
available to us in its own way, we have lost something as well — both the 
certainty and reliability of traditional quantitative data bases and analy- 
sis, and the rich detail that is normally associated' with' qijalitative analy- 
sis. In addition, we run the risk of confusing the reader, who may be 
interested m knowing whether an qbservation is based 6n survey data, case 
data that have t>een coded, or more traditional qualitative analysis techni- 
ques. While we have .<^t tempted to provide a reasonable guide to what data 
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are contritKiting- to specific analyses, full specification would have been so 
tedious that we lave not attcjmpted it. What we hope we have done is to 
provide sufficient detail abput our methods and our data sources tnat re- 
searchers of different methodological persuasions can judge, for themselves 
ho** convincingly our data supported our. interpretations. 

To suftimarize, we have, in this volume, atteiripted^ to blend a variety 
of research traditions. > To the extsnt that we are successful, it is because 
of the many advisors that we have had along the way. Specific mention should 
go to Michael Kane, currently of NIE, ancl formerly of Abt Associates where he 
was the init'ial project director of this study, who had the initial idea for' 
a* "consolidated coding form."' John Egermeier, our indefatigable project 
officer, has provided advice, suppgrt and criticism wKich greatly facilitated 
the conduct of the study. Terry Deal, Sam'^iieber and Ronald Corwin, who have 
served as project consultants, provided us with systematic praise and goads, 
each from his own special perspective. It is \n large measure their compell- 
ing advice that drove us to try and achieve mgny things ih one study. Robert 
Dent let arid Robert Herriott have served as advisors to the study, and have 
reviewed and provided many suggestions about previous drafts of this and 
other reports. Elearpr Farrar reviewed a previous draft oP the volume, and 
•provided, an insightful critique from her own conceptual/empirical perspecj 
tive. Naida Bagenstos and Ward Mason of the National Institute of Education 
also critiqued previous drafts. It has been a pleasure to respond to such d 
stimulating grobp'of colleaguef and critics. ^The final responsibility for 
the docum^jnt rests, of course, with the authors. 

Without the administrative support atyd assistance of Thea Moskat 
throughout the past two and a half years, it is doubtful that we would have 
ever found our data in the ever-growing project files, much less have pro- 
duced a legible report. We also acknowledge the editorial assistance of 
Sandy Margolin, v^hich greatly improved this report, and the able assistance 
of Kathe Phinney in typing this .volume. 

We are especially indebted to the teachers, principals, superinten- 
dents* and other administrators, field agents, and project staff who parti- 
cipated in the RDU program wl^o agreed to have us "look over their shoulders" 
and question them, often at length. They have assisted us generously, 
deapite the valuable time it took away from their primary task of improving 
their educational programs. We ^re grateful for their participation. 
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CHAPTER 1 



DISSEMINATION, KNOWLEDGE UTILIZATION AND SCHOOL IMPROVEMENT 



INTRODUCTION 

This volume explores three interrelated areas of educational policy 
and practice which have engendered persistent debate for the past two 
decades: dissemination, knowledge utilization and school imprpvement. Each 

fof these treats the general problem of creating a climate for "enhancing the 
ef fectlveRBSs of .the educational servaces that are delivered to children. 
Elsewhere (Louis and Rdsenblum, 1981) we have analyzed the ways m whjch t*he 
design and management of mt/erorganizatioriel 'collaborgtive arrangements 
affected the delivery of services to schools and school improvement outcomes. 
Here the focus changea from questions of ^ervitje delivery structures to the 
way m which various intervention strategies and locul contexts affect 
knowledge utilization and * school improvement outcomes. The oyerardiing 

.questions that guide ,^h^is volume are: * 

• To what degree do .various cgmponents of a dissemmat ion- 
based intervention affect knowledge utilization and 
school improvement? 

• To what degree are knowledge utilization and school 
improvement outcomes a fOnction of persistent and 
durable characteristics of the local school and its 

• context? , , 

• Do dissemination-based intervention strategies have 
any effect on local knowledge utilization and school *^ 
improvement activities that cannot be fully accounted 
for by local contextual characteristics? 

On a more general level, these questions can be. summarized as a part 
of the ongoing debate between advocates of planned or managed change and 
those who adopt a more naturalistic systems perspective, which tends to 
argue that the complexities of local organizational structure, climate and 
setting can,. at best, be only partially '^managed." A^ central question in 
this debate^ is, therefore, how much can reasonably be expected of schools 
^that become involved m improvement efforts. At a policy level, these 
'questions reflect the persistent disagreement between approaches that advo- 
cate federal and state support of modest {or even more major) interventions 
m local districts for the purposes of "improvement and those who believe, 
equally firmly,* that local schools- should generally be left alone, and that 
federal and state interventions ai^e as likely to produce regressive conse- 
quences, as positive ones. . 

Indeed, withm the -field of education this entire debate has been 
stimulated by federal efforts over the ^^ast decade to promote educational 
innovation and the dissemination of successful educational programs. . These 
efforts have taken place largely to help bridge the gap. between knowledge 
producers (primarily university-based scholars) and the potential consumers 



of knowledge (primarily practitioners and admiriistraYors at state and local 
levels).., . Without intervention, this gap 'vyould be more likely to increase 
than decrease in the coming years, as the amojjnt of specialized information 
proliferates and access to it by the generalist practitioner becomes more 
difficult. While researchers have bemoaned the fact that their results are 
not used, on the whole little initiative has emerged from the centers of 
knowledge, product ion to deJl^elop sustained communication channels with prac- 
titioners. The academic system does not, 'for the most part, reward effort, 
spent in the application of results, and the press of fulfilling traditional 
role oblvgattbns within the universities militates against individual and 
organizational* efforts to'develop linkages with a wide scope of users. 
Practitioners, too, have fnequentl]^ shown only moderate initiative in 
approaching the knowledge producers for help in finding solutions to a 
growing set 'of problems. This is partly because they do ' not know how to 
initiate such contacts, and partly because the different perspectives of the 
service- oriented practitioner and the research-oriented academij make com- 
munication difficult whgn such contacts occur. » 

Bec3use of the barriers between research and practice, it has become 
increasingly, necessary to develop " alternat ive ways to increase the flow 
of. information from researcher to use: and the flow of needs or problems from 
user to researcher. Historically, the field most active in this area has 
been agriculture, where the effort to develop inst itut lonal izeci contacts 
between universities and farmers began in the mid-19th century. Recently, 
the need for such information flow has been voiced in most prOfes^ons, 
particularly law and medicine. The field of education also has made jnany 
strides in this regard, largely as a result of federally funded efforts. 

The past decade and a half has witnessed a tremendous, growth of fed- 
eral invoivoDent , not onl^ in funding the development of solutions to educa- 
tional lA'oblems in both local school settings and research settings, but also 
in the funding of educational research. As Corwin and Nagi (1972) have 
pointed out, the increase in the research budget of the U.S. Office of 
Education (USOE) from the mid-50s to the mid-60s was one hundredfold, and the 
trend continued -until the mid-70s (NIE, 1976). *This funding of research, 
predicateo on the assumption that knowledge is useful and will eventually 
result ip a payoff to the larger society, has tended to be directed more to 
ap^plied and developmental research than to basic * research. 

Since the ex istence of research has not always resulted in the 
, increased use of research, the federal efforts have included the development 
of a'variety of mechanisms designed explicitly to stimulate the use of 
innovations. In the late 1960s "centers of excellence"— R&D centers and 20 
regional edocitional laboratories — were created to orient research activities 
towards developing educational innovations. Today the surviving centers* and 
the regional laboratories have, as part of their mandate, the goal of dif- 
fusing the .products they develop (NIE, 1979). Another major effort towards 
utilization of information was the creation of a cofuputer-based cataloguing 
system called "Educational Resource Information Center" (ERIC)', implemented 
through a ^series of 16 clearinghouses (Thompson, forthcoming). 

Such structures are admirable in concept and have l^un to attack 
the core goal of producing and disseminating usable inforlnat ion, but they 



probably have not had a direct impact upon most American schools. Gideonfee^s 
(1970) reviNew of research and development efforts in the United States 
concluded, for exa!T)pl.e,' .,that lack of innovation characterized the curricula 
m uver half the pchools m the country, while the National Institute of 
Educptiofi (NIE) supported this conclusion with aifferent data (I^IE, 1973). 
Our own data,- presented m Chapter. 8, suggest' that this has not changed 
signif'icantly m the past decade. 

Moreover, a study of innovations in urban high schools (Nelson 
and Sieber, 1976) found that many schools were adopting innovations that 
were considered to. be of low quality by educational* experts* indicating a 
need fiot only for improved disserT\ination processes, but als9 for improved 
quality cbntrol af dissemmatable innovations. Of equal concern is the 
adoption of high-quatity innovations * in inappropriate situations (C£j.rter et 
al., 1976). ' » , # 

It, also became clear that, the ERIC system, while highly acclaimed 
by practitioners as a valuable resource, may not* be reaching a broad audi- 
ence. An evaluat;ion survey of ERLC users in 1970 by Frey ('1972) found that 
62S» of "the users . were college students and only 21% of the users were prac- 
ticing teachers.* In addition, a recent NIE report indicated that; the level 
of effort devoted to dissemination by the centers and labB was generally low 
4NIE,-1975). Thus, additional avenues for the dissemination and utilization 
, of educational knowledge and practices may be necessary. 

Jhe educational R&D system, including ERIC* and the labs, has been 
faulted for its level of effort in disseminating research and new ideas 
to pracjt it loners *( see Thompson, 1981). In^ recent years, however, consider- 
,ably more focjjs has been devoted to dissemination and regional sfer.vice in the 
regional laboratories, as is exemplified by the Regipnal Development and 
Exchange programs, and the Cooperative School Improvement Program, both of 
which . involve cross-lab -cooperation xn delivering regional dissemination 
services.* Products of the ERIC ^system may also be in wider use, as a conse- 
quence of better dissemination practice's m state departments and *inter- 
mediate education agencies (Royster et al., 1981). Others, ' however , have 
noted that the problems dp not Test entirely with the "senders" such as labs 
or universities. , Rather, the incentive systems in schools do not 'necessarily 
reward the use of information or program change at the school level (Sieber, 
1981), and the most cogent of messages may, therefore, fail to be heard. 

FEDERAL AND STATE STRATEGIES FOR SCHOOL IMPROVEMENT 

As a consequence of the increasi/ig awareness of the ^gap between 
research on educational improvement and change in schools, federal and state 
agencies have been increasingly involved in legislation and programs to 
stimulate innovation in schools. Several basic non-judicial federal/state 
strdtegiies in stimulating local school improvement efforts have characterized 



*It is important to note that many college student? are future 
teachers,* and some (unspecified) proportion of them may be actively teach- 
ing.' There is no reason to think that direct access of ERIC b^ teachers has 
increased over the past decade, however. 
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•the past two decades, amopg the most important of Which are a legislative 
mandate strategy and a ^direct 'support strategy. 

The legislative mandate strategy is, perhaps, the best known federal 
and state intervention mode, and- is exemplified by legislation such as PL 
94-142, which mandates individualized educational plans for handicapped 
children,, or the sex equity provision m the Vocational Education Act ^f 
1976. In these cases the legislatures and executive branches have produced 
laws and regulations which require distripts that seek to receive funding 
from fefleral or state resourpes to meet certain specified standards. The 
basic purpose of the lawL is not to create "innovation" or "knowledge utili- 
zation" m schools. HowevQ?*, much change may be st imulated m order to ensure 
that the monies available will be applied m settings which conform to 
• legislative standards of equity, quality, or efficiency. Thus, the legisla- 
tive mandate strategy is based on the observation that, if the value c. the 
aid IS large enough, and the changes required are within, reasonable grasp of 
a local district, then legislative standards will stimulate districts and 
schools to change their behavior, ¥here necessary, to conform. In the 
process, knowledge utilization and school improvement may (aid according to 
practitioners frequently do) occur. ^ 

V 

Since the mid-60s, a second, and widely used strategy for stimulat- 
ing improvements has been for the federal or state government to provide 
support' for innovative activities. Support may be of *three types: fiscal, 
technological-^, or hyman. 

Direct fiscal support strategies .iiay range from "seed nvoney". programs 
(short-term funding directed exclusively at promoting innovations in curricu- 
lum) to demonstrations . to more permanent formula funding which has implicit 
lUid explicit guidelines that require innovation and change in order to meet 
program guidelines. For example, both the USOE and the NIE have been in- 
volved m* the design of programs whose intent is^ to provide "seed meney" 
targeted to the improvement of educational problems m schools. Among the 
programs that rely on such a strategy are those that have recently been 
studied by Berman et al. (1974; 1975; 1977), such as- Right to Read, Title 
III and Title IV Bilingual Education. According to many school administra- 
tors, even Txtle I, which was iritended almost exclusively as a compensatory 
t:)rograin to target disadvantaged students, has been a major source or stimula- 
tion for program improvement, and has benefited students who do not directly 
receive Title I services. A more directed approach to the "seed money" 
strategy may be found m the NIE-sponsored Rural Experimental Schools pro- 
^^am, m which participating school districts were asked to plan and imple- 
ment locally developed programs for "comprehensive change" that touched upon 
all areas of school functioning (Herriott, 1980)^. 
* * 

The strategies which are the focus of this research do not provide 
support directly to an innovating school or district. Rather, they attempt 
to develop a support structure that is outside the LEA, but which provides 
either technological or human support for school improvement. This approach 
has been a mote recenb development m education, jjut is growing very rapidly. 
For example, the growth of intermediate service agencies,^ which are typically 
supported at least in part by the state, has been enormous over the past ten 
years and shows f^w signs of abating (Ym and Gwaltney, 1981; Stephens et al. 



1979). The major function of intermediate service agencies in most states is 
to provide both technological (information retrieval, and other progran 
information) support and support in providing staff development and training 
programs to supplement those that can be managed by local districts in their 
serv ice area.* 

The fiscal support strategy emphasizes innovation, but not neces- 
sarily the utilization oi^ existing validated products that have emerged 
from R&D in education. The technological support strategy utilized in 
federal and state funding has been to encourage the adoption of existing 
programs througn prod'-.cts pf various types. Recent efforts of this type 
include the following examples: 

The Office of Education's sponsorship of the packaging 
cf exemplary programs for dissemination and replication 
( PIPS ) . These consist of a small number of compensatory 
programs (the first set consisted of six programs) which 
, have been locally developed, evaluated and approve^ 

through the Joint Dissemination Review Panel (JDRP) 
(Stearns et al., 1976; Campeau et al., 1980). 

• The Pilot State Dissemination Project , inaugurated in 
1970, in which seven field agents in three state-based 
organizations aided school practitioners in solving prob- 
lems by drawing upon existing research information (Sieber 
et al., 1972). 

• The National Diffusion Network (NDNJ, a national dissemi- 
nation system established by the Office of Education in 
1974, in which a limited number of "developer/demonstra- 
tors^* (DDs) of locally developed programs approved by the 
Joint Dissemination Review Panel (JDRP) provide a limited 
amount of technical assistance to schools in the adoption 
of 8uch programs. The DDs are "linked" feo schools by 

• "state facilitators" in 40 states and provide support 
services to a large nunber of schools (Bmrick, 1976).** 

• The State Capacity Building Program , established- by NIE 
in 1975. to build comprehensive disseminat ion capacities 
within states. This program in 23 states focuses on 
building a comprehensive set of knowledge resources, • 
developing means of linking educational client groups 
with the resource base and developing leadership and 



^Another major function, and the original motivation behind Jthe 
creation of ISAs in m^ny states, is to coordinajte many administrative 
functions that are performed mole cost-effectively at a supra LEA 
level. However, ev«n when these are the major purposes, indirect support 
for school improvement activities i§ also present m most cases. 

♦♦A nrajor - study* of NDN and other federally funded dissemination 
activities i* presently being ^completed by the NETWORK, Inc. undqr funding 
from the Department of Education. 
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management arrang«Dents which facilitate the provision of 
such services (Royster et al . , 1981). 

• RAO Ut ilizat ion Program funded by NIE in 1976, in vyhich 
seven projects (four SEAs and three consortia) provided 
an intensive grray of support services through linking 
agents serving approximately 300 schools. The projects 
supported the change proces. from problem identifica- 
tion to implementation of solutions and incorporation 
of innovations, constructing necessary linkages to a 
validated R&O outcome. 

*^ 

Most of the above programs also incorporate human support, through technical 
assistance to facilitate effective product choices, implementation or both, 
however, a few- programs have placed greater emphasis upon the human assis- 
tance strategies than on the teclmolog ical-. The chief among these is the 
Documentation and Technical Assistance Program (DTA ) wh ich sought to assist 
schools m selected urban areas: 

• DTA was funded by NIE to develop a knowledge bap? that 
focused on improving organizational process m schools. 
It emphasize^d^ pull ing together '*craft" knowledge and 
raking it available to other schools through face-to- 

^ facen, i>ntensive assistance based on organization devel- 

opment principles (Miles, 1980). ^ 

For these and future progr kirs-^to be successful, moie than just 
the dissemination or utilization process neette^to be understood. The objec- 
tive of most research and development ef forts,^-^and the subsequent use of 
these efforts, is educational unprovement. Educational improvement usually 
occurs through change i., the structure and/or process of educational systems 
leading towards increased effectiveness in achieving individual and organiza- 
tional outcomes. Therefore, the issues of dissemination, utilization and 
achievement of outcomes can be more productively examine'i, and recent' efforts 
more meaningfully discussed and compared, by using an educational change 
conceptual framework. However, before turning to the task of explicating a 
model for understanding dissemination, utilization and school improvement, we 
must first achieve a better understanomu of the meaning of these terms 
as we use them. 

Dissemination, Knowledge Ut il izat ion and School Improvement ; 
Clarifying the Issues 

As the above discussion indicates, thnre is concern in both the 
research, practice and policy communities over the three issues on which this 
volime focuses. However, despite the attention given to them, they are 
often referred to interchangeably, and inadequate distinctions are made 
between a dissemination focus, a knowledge utilization focus, and a school 
improvement focus. The lack of attention paid to defining these terms 
more, prec isel y , and showing how they are interrelated has led, in many 
cases, to debates o »r non-issues, or to major disagreements over poli- 
cies that are not incompatible* - In addition, to laymen the confusion of 
terms and their interchangeable use sounds faddish and full of jargon. Our 
approach to defining the terms will not attempt an intellectual or policy 
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history, in large measure because, despite the contentions of some policy 
researchers, we are not convinced that there has been any clear evolution 
from ooftf focus to another (Thompson, 1980). Rather, each focus has a dis- 
tinctive history, and each has interas^ted with the others in a variety of 
ways over time. 

Disseminat ion . The term dissemination implies a focus upon a sender 
of informetion. Whether dissemination is impersonal (senaing out information, 
^sing telev ision spots) or two-way (involving intensive contact between a 
change agent and a receiver) the emphasis in dissemination is on the process 
of supporting the spread and exchange of information. According to the 
recent report of the Disseminat ion Analys is Group ( 1977 spread refers to 
"the one-way casting out of knowledge in all its forms" (p. 3), while ex- 
change IS defined as "the two-way or multi-way flow of information, products, 
ideas and materials" among a variety of actors (p. 4).* 

The notion of dissemination implies a variety of policy/management 
questions, and a number of research or theory questions. Some of the manage- 
rial and policy issues relating to dissemination include the following: 

It Since an agency or an individual is engaged in dissemina- 
tion, what IS to be d issemiriated*^ 

• What criteria for determining the quality control pro- 
cedures should be used in developing a base of informa- 
tion to be disseminated'^ 

• How can systems be designed and implemented to most 
effectively retrieve relevarif information from a knowl- 
edge D' e'^ 

• Wha^ c es are best placed to engage in dissemina- 
t ion t-c*' ivit les*^ 

• What kind of information can be most effectively dissemi- 
nated'? 

• What mechanisms of "sending" should be used*? 

These issues have been in existence since at least t^e mid-60s, yet are far 
from being resolved. Inus, for example, despite the enormous effectiveness 
ofl the Educational Resources Information Center (ERIC) system, there are 
continuous management questions raised about how to screen, catalogue and 
store the ever increasing number of items which ERIC includes. Similarly, 
although the National Institute of Education commissioned i review of the 



*The Dissemination Analysis Group report includes a broader defini- 
tion of dissemination. However, we believe that the broader definition 
offered by this group muddies some distinctions in conceptual focus which are 
important. We therefore include the final two "levels" of dissemination used 
in the DAG report — choice and iinplementat ion--as part of knowledge utiliza- 
tion rather than dissemination. 
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Joint Dissemination Review Panel's procedures for certifying programs which 
was designed to broadly publicize the criteria by which new programs are 
judged '(Tallmadge, 1977), m recent years some concerns have been raissd by 
developero and practitioners about whether these criteria are always appro- 
priate, and whether they are so burdensome as to discourage the inclusion of 
genuinely jexemplary and innovative programs (Miles, 1980). In summary, there 
IS a dilemma as to whether quality control issues should drive a dissemina- 
tion system. Indeed, several of the reports that hSve been written as part 
of the Study of the R4D Utilization Program have focused explicitly on issues 
of dissemination (Ym et al., 1980; Louis and Rosenblum, 1981), as does this 
volume. 

The theoretical issues underlying a dissemination perspective are 
well explicated, and focus on basic issues of interpersonal communication. 
What kinds of messages will be attended to by different receivers? In what 
ways, and to what degree do the mechanisms of transmission affect the 
impact on the receiver? What are the characteristics and behaviors of 
effective change agents or other transmitters of information from one group 
to another*^ ^ 

The term dissemination has recently come into some disrepute among 
a variety of policy analysts. It is often identified with a "technological 
perspective" on the process of social change, which is viewed as primitive 
m its explication of and attention to the social systems in which dissemina- 
tion occurs (House, 1981). Others. have attacked a dissemination focus 
because it does not attend adequately to the long-term effects of information 
after it reaches a receiver (Berman and McLaughlin, 1974), or even to the 
potential user system's need for information (Knott and Wildavsky, 1980). In 
its more extreme forms, "dissemination" has been a s s u med rflso to be imperi- 
alistic and mechanistic, attempting to squash local variations and impose 
externally determined modes of behavior (Miles, 1980; Thompson, 1981). As 
this volume will show, however, such issues are quite salient in almost any 
effort to develop a program of change for schools. 

Knowledge Utilization . If dissemination focuses on the sender and 
sending of information, knowledge utilization emphasizes the user and user 
system. Knowledge use is not limited to the ways m which potential recipi- 
ents of information from a dissemination activity use that information nor is 
.it limited to how they use "external" mformatiog. Among the earliest 
programs supporting knowledge utilization, for example, were the School Study 
Councils, active during the 50s, which emphasized internal research and 
analysis (Dannenburg, 1970; National- School Development Council, 1979). 
Knowledge use can be viewed as the study of generic problems encountered when 
individuals and organizations look for, react to, create and make use of 
knowledge or information. More broadly conceived, it is often Viewed as 
incorporating the study of planned change, smc^e planned change activities 
invari£4bly include "using knowledge." 

In recn^t years, interest m developing policy r.elated to knowledge 
use, and in studying knowledge use in education has burgeoned, perhaps 
largely as a result of a number of studies that emerged m the later 60s and 
early 70s which indicated that actual change in schools as a result of 
claimed "knowledge use" was often limited Tuross et al., 1971; Charters 
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and Pellegnn, 1973; Gideonse-, 1970). A potpourri of policy and management 
issues that represent enduring but unresolved problems are: 

• To whaL degree should adaptation of developed products be 
encouraged or proscribed? 

• How can policy or program management improve the choices 
that schools and educators make m choosing new program- 
matic efforts? 

• How can extern?lly developed programs foster better 
planning and implementation activities m schools'^ 

• What IS the appropriate level (classroom, school or 
district) to encourage and support knowledge utilization? 

These questions, ^and others about knowledge use, reflect a basic 
concern about whethef the process of knowledge us% can be better defined, 
predicted, and assisted* In each of these areas, some progress has been 
made, but more is needed. For example, there is clear consensus that knowl- 
edge use IS not a rational, dichotomous" event as was often assumed in the 
recent past. Rather, knowledge use tends to be viewed as^ a cognate of 
personal or organizatjcna^ change, and as a relatively long-term process that 
can be viewed m stages. Among the best known versions of a stage-of-use 
approach m education is- the~"wcrrk— Gen© Hall and his colleagues at the 
University of TeAas (Hali et al., 1975), but early versions are apparent in 
applied psychology beginning ^^itS the 1940s (see Havelock, 1969,- Chapter 3, 
for a review of early research). Because knowledge use is a complex personal 
and organizational process, there are many different types of use ranging 
from implementation of new programs on a district-wide basis to routine, 
everyday use of knowledge by teachers m classrooms (Huberman, 1981). Again, 
this notion o*' multiple types of use is not novel, but has achieved new 
currency as a consequence of recent work which has attempted to account for 
the lack of "implementation" of research findings in policy (Weiss, 1980; 
Lindbloom and Cohefi, 1980; and Pat ton, 1979). Other issues which have circu- 
lated for years, both m education and general orqaniz^ional literature, 
concern the effects of participation m decision makfhg on ate at the indivi- 
dual and organizational level. ^ 

It IS iniportant to emphasize that, while knowledge use has received 
a great deal of attention, the conceptual issues are far' frbm resolved. 
For example, over the pust ten years, knowledge utilization research has 
cycled away from an efmphasis upon cognitive or personal change, to'ward 
an emphasis upon implementation of new programs at the organizational level 
and currently the emphasis is upon cumulative adjustments between knowledge 
and individual or organisational behavior. Although conceptual clarity is 
not yet apparent, one thing is clear: knowledge use has changed from a 
status as the dependent variable m a study of dissemination or diffusion, to 
an arena of study m its own right. 

Schoo l Improvement . In the field of education, school improvement 
may be thought of as the implicit objective of both dissemination and knowl- 
edge utilization. It IS also, however, an area of programmatic and research 



activity with a distinct focus of its own. First, while a school improvement 
focus emphasizes 'improvements in student outcomes it often has a broader 
meaning which encompasses not only improvements in jpurr iculum and cognitive 
achievement, but also covers expanding the school)© capacity—capacity for 
self renewal, for innovation, and for knowledge utilization (Miles and Lake, 
1967)'.' Thus, school improvement tends to fociis not only on the technology of 
teaching^ but also on the incentive scructurp, the distribution and' availa- 
bilifcy of resources, and school climate ami staff development activities, 
^he notion of school improvement is often re formist in intent and approach. 



Rather than assuming that better schools cah result from rearranging the 
various teaching tasks or from broader availability of curriculum, school 
improvement often commits itself to changing the '^tructur^ and decision- 
making patterns in schools, and to making basic changes in the degree to 
which school personnel believe they can take responsibility for their own 
activities. For example, the 1973 NIE report. Building Capacity for Renewal 
and Reform , makes the* following statement: ^ 

Because of the diversity within ttie operating system and 

the decentralized decision-making processes, reform and 

renewal of local school districts will occur oijly if those 

districts develop the capacity to be more analytic in 

their behavor, more sophisticated in the choice and use 

of resources .. .and better able to assess critically the 

effectiveness of what they are doin^ (p. 9^). ^ 

A basic assunption underlying the schooL improvement focus is that, in order 
to provide an effective climate for learning, schools rhust be "healthy" 
organizations. In addition, it is often emphasized t+iat 

Real improvement in learning is determined not so much by the 
adoption of specific educational practices as by tte modi- 
fication of organizational conservatism . . . and organiza- 
t onal pathology (Schmuck and Miles,- 1971, p. 1). 

Programmatic thrusts with a strong school improvement focy^^in 
the broadest sense have more often come from agencies not associated with 
state or l^ederal qovernrnqnt. For example, the League of Coopej^gtii/e Schools 
was sponsored by a nonrprof^ organization called I/D/E/A. Individually 
Guided Education (IGE) wjrtcn is officially designated as an "RAD c^t^iculum 
innovation" is, in factaC^a reformist program for restructuring the entire 
school to be more congruent with school improvement ideals. However> the 
federal government has sponsored some activities of this nature, typically 
with the goal of achieving school improvement through a managed change focus 
and/or improvements in the knowledge utilization capacities of schools. 
Perhaps most ambitious among these activities has been the Docunent at ion and 
Technical Assistance Project, sponsored by the National Institute of Educa- 
tion, and described btiefly above. 
*-> 

The emphasis on school improvement is,, however, rarely an end in 
itself. Rather, there are clear assumptions underlying this focus that, in 
order to improve the delivery of services to students and to improve studeot 
cognitive outcomes, schools must become more vital, self-sustaining organiza- 
t ions. 
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COMBINING THE THREE FOCI IN PRACTICE: THE RAD UTILIZATION PROGRAM 

In June 1976, the National Institute of Education (NIE) established 
the Research and Development Utilization (RDU) program as e new iaction- 
Research effort in dissemination. One of the major propositions that the RDU 
program was designed to test was whether school-level practices could be 
improved by making external resources available to school personnel. Over- 
all, the program strategy was to: 

• organize a linkage system whereby national, state and 
^ other external resources would be maoe available to 

school personnel (dissemination focus); 

• apply research-based products or ideas to locally defined 
school problems (knowledge u^e focus); and 

• develop a pfoblem-solving process whereby schools would 
systematically identify such problerps and select and im-^^ 
plement new ideas (school improvement focus). 

The RDU program is unusual among federally funded dissemination 
strategies^ because of its commitment to the dissemination and use of R&D 
products and the development of local school capabilities to solve prob- 
lems through the use of externally generated knowlAge. Other federal 
programs have tended to concentrate on either product Wssemination or local 
capacity building, but have not concentrated 6n an integrated model for 
combining them.*" The core of the RDU strategy was to .provide eadti partici- 
pating site, either a school or a district, with assistance in the following 
sequence of activities: ^ 

• identification of a problem or set of problems; 

• examination of alternative solutions to the problem, 
4 focusing particularly on the products of educational 

research and development (R4D); 

^ ' • selection of a specific solution' t6' address the problem; 
m implementation of the solution; and. 

• evaluation and incorporation of both the solution and 
the problem-solving process. 

The service delivery system of the RDU program operated through 
seven projects, each of which poordinated a network of organizations and 
individuals that were involved in the provision of services 'and information 
to local schools and districts. Although the seven projects varied in 
Structure and design, most comprised four types of organizational units: 

t 

♦The best statement of the assumptions underlying the program is 
found in Hutchins, 1975. 




A headquarters unit serving as the formal recipient 
of the fedeJ(;al award and aa the general administrator 
of the rest of the network; four of the seven projects 
operated out of state departments of education and 
comprised a linkage system within a single state, one 
operated out of a state department of education but 
served a four-state region, ^and two operated out of other * 
types of organizations (a non-profit educational R4D 
jcenter in one case apd a national association in the 
other) creating linkage systems dispersed across the 
entire nation; * , - . 

• Resource orqailizat ions , often university-based or inde- 
pendent organizations that had developed their «own 
expertise in educational R4D, training, and technical 
assistance* One function that was typically carried out 
by a resource organization was the consolidation of a 
"knowledge basej^'^ or pool of R4D products, developed as 
resources for identifying solutions to match client 

" ' needs; ^ ' 

• Linkage organizations , usually an ilptermediate service 
agency or state educational agency, employing "field 
agents" who coordinated the services provided to local 
schools and districts, and who helped guytde the local 
school personnel in a school improvement process* 
Each projert supported two dt more field agentd; and 

A 

• Local school districts or schools which were responsible 
for engaging in a problem- solving process culminating 

in the adoption and implementation of new practices based 
on the "external knowledge"; each site typically estab- 
lished a local team of teachers and administrators, and 
with, the a^^istance of the field agent, gener^Jly made 
* major decisions related to the schoot improvement effort, 
thus fostering Idfcal ownership of the program and the 
selected solution. 

The headquarters unit of each project developed a set of formfel 
relationships, usually reflected in some sUb contractual agreement, ampng the 
major participating .resource and linkage organizations; formal agreements 
were al£o struck witfjf participating school districts. 

The network components were typically organized. into a linked struc- 
ture of horizontai^functions , cooidinated and/or - conducted by the project 
headquarters unit, and vertical linkages through which the RDU project 
operated (see F:^gure 1-1)^ The horizontal functions included project manage- 
ment, and the work of^the resource organizations which developed and^main- 
tained a specific R&O product base, provided training and technical as- 
sistance, and oonducted proje^^t evaluation and related research. The 
v^ertical linkages actually included some type of communication (eithef weak 
or strong) among six potential ad?Rinistr stive levels tl^ouqh which the RDU 
project^j operated or del iveredf services. 
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The seven RDU pr<?ject8 were regionally distributed, and included the 
following: 

The Northwest Reading Consortium , involving the state 
* departments of education pnd other agencies in Washington, 

, Oregon, Alaska and Idaho; 

• The National Education Association Inservice Education 
Project , operated in collaboration with the departments 
of education and corresponding state education associ- 
ations in 12 states: Alabama, California, Iowa, Massa- 
chusetts, Michigan, Minnesota, Ohio, Pennsylvania, / 

■^Tennessee, Washington, Wisconsin, and Wyoming; 

» 

•i The Consortium , operated by The NETWORK, a non-profit 
research and service organization that coordinated the [ 
efforts of agencies in siy dtates': California, Connecti- 
cut*, Kansas, Massachusetts, Minnesota and Washington; 

• The Georgia Research and Development Utilization Program ; 
^ t The Pennsylvania School Improvement Program ; 

• The Florida Linkage System ; and 

• The Michigan Career Education Dissenvination Project . ♦ 
This project was operated by the state department of 
education as were the projects m Georgia, Pennsylvania 

.and Florida. 

All of the seven, projects have completed the federally supported 
service delivery part of their activities. As a whole, the seven projects 
operated m 20 states and served over 300 schools or school districts over "a 
three-year period (1976-1979). Over 90S of the local sites which'became 
involved in the RDU program successfully completed the problem-solving 
process. Of these sites, 80^ ad^^pted and implemented a research-based 
new practice under the aegis of the program, and mo&t of these were received 
with enthusiasm m the schools. Thus, when viewed from the local site per- 
spective, the networks that were created to help schools improve local 
practice can be considered a success; however, many of the projects faced 
significant problems in establishing themselves and in carrying out their 
functions (Louis and Roseftblum, 1981). 

Overview of the Study of the RAD iftilization Program and School Outcomes 

The RDU program, with its emphasis upon dissemination of tested 
information and the . process'^of knowledge use in the local school, "and its 
attention to broader school improvement functions in local schools presents 
an ideal opportunity to explore the way m which these different components 
interact ih increasing the effectiveness of schooling. 
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The Study of the RDU Program ^ / 

In Novent)er 1977, Abt Associates inc., a social science research firm 
based m Cambridge, 'Massachusetts^ contracted to conduct a study of the RDU 
program. The study addressee six major issues: 

o how relationships are, managed among the various agencies 
thit possess the* expertise and cesmijrces to help local 
schools solve problems; 

o to what degree, an interventio-i- program such as ROD can 
help schools overcome barrieis to successful problem 
solving. (such as limited access to information or lack 
of planning skills, etc.); 

JO to what degree the products of educational R4D are 

relevant ttv^the problems and contexts of local schools; 

o what the impact is of the products of educational R4D 
' once they have been adopted and implemented; 

o what factors contribute to tb« institutionalization of 
the RDU approach withm a variety of organizations; 
and f 

o how linking agents coordinate the flow of external 
resources to schools, and whether this helps the 
schools solve problems. 

Ddta were collected m face-to- face focused but unstructured inter- 
views at 51 sites from 1978 to 1980/ Case studies were written on 46, 
sites, five of which also received site visits. We also conducted mailed 
surveys of principals and a sample of teachers at participating schools 
during the fall of 1979'. Additional descriptions of our general methodolog- 
ical approach are presented m Chapter 3. 

Our study of the operations and elements of the R4D Utilization 
program at the school level has had a number of component a, of which this 
volume IS only one. Two other reports have been directed at an audience 
jof local administrators and practitioner's, and hav^ attempted to synthesize 
our observations about effective ways of building capacity for school 
improvement through the use of multi-constituent problem-solving groups 
(Kell and Louis, 198^1; 4.ouis, Rosenbluni and Kell, 1981). In another report, 
case materials illuminating broad problems of managing change within scnools 
are presented, along wilh an analytic context for assisting local personnel 
m understanding the prob^en^ of fl^^nagirig a knowledge utilization and school 
improvement process (Louis, Kell, Chabotar and Sieber, 1981).* In this 
volume our focus is more firmly grounded m the policy questions which 
stimulated the development of the RDU demonstration: e.g., the impact 
of dissemination strategies upon knowledge utilization and school improve- 
ment. ^ * ^ . 



*An annotated bibliography of «^11 reports from this study may be 
found m Louis and Rosenblum (1981b). 
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Our general approach to this volume has been to develop a theoreti- 
cally ancf^empirical ly grounded schema that attempts both to categorize 
the "strategies" employed by the RDU projects into different types^ — informa- 
tion , technical assistanc^i and a problem-solving process — and i\ examine 
the effects of various naturally odcurring variations within these strategies 
upon a variety of intended and unanticipated potfentiaJ outcomes. This 
schema also incorporates the assumption underlying the natural systems , 
school improvement focus, that local site characteristics condition and 
interact with any externally ' stimulatnd inte^ention and local decision- 
making behaviors, and have, therefore, both a direct and indirect Impact upon 
any observable improvements or changes. The schema, which "Is further ^expli- 
cated in Chapter 2, is based not only upon empirical observations of ongoing 
activities in schools, but upon the brdad set of literature regarding dis- 
semination, school improvement and knowledge use. 



^ CHAPTER 2 

STRATEGIES FOR CHANGE AND SCHOOL IMPROVEMENT: A GENERALIZED MODEL 

\ IN^rRODUCTION 

The^RDU program consisted of a combination of strategies for school 
improvement that emphasized dissemination of tested information, the provi- 
sion of technical assistance and training, and a series of problem-solving 
activities in local schools. In order to develop a general' model to guide an 
exafiinatibn of the effects of these strategies on school improvement outcomes 
we have consolidated the resulls of: 

• a review of approaches to the study of educational change; 

• aft elaboration of the nature of the RDU intervention str^- 
* tegies and their hypothetical relatiorlship to each other 

and* to school, improvement outcbmes; and' 

• initisl empirical observations of activities in the RDU 
projects and schools* 

Such a consolidation was necessary for several reasons. The RDU 
program represented ap ambitious and complicated effort. In it were meldod 
Several intervention strategies, not' all of which were fully . articulated at 
the time of the program's inception. Thus, the first s^eps in our inquiry 
were to specify the nature of the strategies that characterized the RDU 
intervention and to array these empirical descriptions of RDU services within 
a frimework of existing theories about how best to produce change in schools. 
In the remainder of this chapter, we review some of ^the dominant approaches 
to the study of . educational change, and we present a model that shows how the 
RDU program may be understood within that ciontext. We then discuss how the 
remainder , of this volume addressed the issues raised by the model for an 
investigation of impacts of RDU upon school improvement and knowledge utili- ' 
zation outcomes. 

APPROACHES TO THE STUDY OF EDUCATIONAL CHANGE 

The .proliferation of both theoretical and empirical studies of 
educational change indicates that this topic has become one of the most 
populai research issues in education today. At present, however, we are 
beginning to see the emergence of two distinctive streams of research, each 
of which is characterized by certain strengths and def icieiteies. Jhe two 
approaches may be called the "Strategies of Change" perspective and the 
"Innovative' Organizations/Natursr Systems" perspec;tive. These are discussed 
briefly below by showing t(je commonalities and differences between them.* 



♦Ci^rly- the two perspectives do not encompass all current theory 
aBout organizational change. However, they do represent two of the most 
proJhinent perspectives. For a discussijQn of a broader set of theoretical 
viegwpoints on edacational change";' see Lou^.s, Kell, Chabotar and Sieb^r 
(1981), or Clark, McKibbin and Malkus (1981). 
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StrateqieS'Of-Chanqe Perspective 

^ Recent research and theory associated with th^ strateqies-o change 

perspeptive are directly linked fo the issues apd problrfns of dissemination, 
knowledge utilization and school. 'improvement: The emphasis in this research 
is on varying apprijaches tj stimulating change within Schools, particularly, 
through planned organizational change (Bennis, Benne and Chin, 1969). What 
sets this tradition apart from those studies ana models that we have classi- 
fied as "innovative organizations studies" i%|the emphasis on how external 
and. ^internal actors, in concert, may affect change. 

One of the precursrfrs. of the works within this group Was the review - 
of the literature relevant to the problem of dissemination and utilization in 
education produced by Havelock (1^69). This review provided policy m'Skers 
and ree^drchers with b cor^endium of concepts and variables related to the 
development of a system in education modeled after the agricultural extension 
program. In such a system, human agents would assist potential adopter^ of 
externally developed innovations by transfating research/results into sugges- 
tions that were suitable for the adopter. The three traditions in the study 
of dissemination discussed by Havelock (and some subsequent ^ react ions to 
4 them) include: 

# The Social^ Interaction Perspective . This perspective 

focuses on the adoption of specific new practices by indi- 
viduals. The explanatory variables most frequently exam- 

* ined include individual characteristics of the ^adopter, or 
the individual's relations within a 'social network of 
peers (Rogers and Shoemaker, 1971; Coleman, Katz and 
Meqzel, 1966; Carlson, 1965). This body of literature 
provides a sophisticated set of concepts and variables 
which may be. related to the adoption process, but provides 
little insight into factors outside of a delimited social 
system which might influence innovation. Another limita- 

^ tion of studies of the diffusion of new practices is that 
they have been almost exclusively concerned with the adop- 
ta|)n of a single, highly visible product such as a new drug, 
or a new hybrid corn. 

• The RDDU Perspective . The "research, development, 
diffusion util ization" '(RDDU) model' is derivef! from the 
vertically integrated* systems of R&D ,found m highly ra- 
tionalized military and industrial concerns (BrickeM,* 
1964; Havelock and Benne, 1969; Cuba, 1968). The adopter, 
whether s/he is a member of the same organization ot i^ 
Ideated in another context,- is viewed as rational; when 
presented with a "good" innovation, s/he will tend to use 
it. In addition, s/he may also be seen as powerless to 
resist following suggest ions* even if s/he does not immed- 
iately perceive their benefits (McDonald, 1971; Sieber, 

'1972). Such. a marketing- model is essentially one^of 
bureaucratic innovatit)n, where those at the top. a"re 
assumed to know what the client practitioner should be 
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doing. ' The macro view of the innovative process is useful 
m drawing attention to all the institutions that ,mjst 
interrelate in order to develop a dissemination system. 
It suffers, however, from a bias tbward* the* needs of 
administrators of such programs, and a view of the practi- 
^ tioner as a "helpless functionary" (Sieher, 

• The Problem Solver Perspective . . The problem solver per- 
spective has the strongest theoretical base of the older 
traditions, emerging from the work of Lewm and the re- 
search on group problem solving pioneered by the Institute 
for Social Research and the National Training Laboratory. 
The problem-solving model focuses on the individual or 
group that is in the-process of changing (Lippitt, Watson 
and Westley, 1958; Fullan, 1972). The change process is 
jeen as a series of stages. The number of stages articu- 
lated by researchers within Ihis tradfTion varies widely 
from study to study, but usually begins with a user recog- 
nition of a need for change, followed by some examination 
of alternative Solutions' to the problem, and thje 'selection . 
and implementation of' a solution. While this tradition 
has often been criticized for its highly rational approach 
to change (Berman and McLaughlin, 1975), the approach it- 
self does not assume moiv.idual rationality, but father it 
assumes that improved problem-solving* processes will 
increase the match between the actual problems and the 
implemented solution. 

Havelock (1969) has noted that each of these traditions is, in 
itself, inadequate as a model for either the design or study of systems to 
increase information use or the adopt mi i ne.w practices^ in education. 
He proposed a synthesis of the three approaches, and the development of a 
single linkage- model that incorporated elements from all earlier models. One 
of the. main features in Havelock's synthesis was a prescription to involve 
the client system in defining the need or problem that required information. 
He pssumed that, a system of linkages which involved the client as a partner 
m the dissemination and utilization process would be. most likely to result 
in change and actual use of research results. Another main element in the 
linkage model is the role of a field or linking agent . Field agents have 
been incorporated into a wide range of programs designed to improve social 
services in many fieldp (Rogers et al., 1976; Glaser 'and Becker, 1974). 
Individuals ir. such roles can contribute to improved problem solving in 
several w^ys, including: 

• facilitating the transfer of information; 

• delivering technical assistance; * 

^facilitating the decision-making process by clarifying 
/goals and providing leadership; and 

• mediating among autonomous and sometimes competing organ- 
izations whose resources^ and services must somehow be - 
coordinated. 
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Some of' the major empirical works within the strategies of change 
tradiikion that postdate the original Havelock synthesis are Havelock's 
extensions of the linkage model (Havelock and Lingwooo^ 1973; Lingwood and 
Morris, 1976), the development of the Concerns Based Adoption Model (CBmM) at 
the University of Tcos (Hall, 1974; Hall et al., 1973) the Study of the 
Pilot State Dissemination Program (Sieber, Louis and Motzger, 1972; Louis and 
Sieber, 1979), the evaluation of the National Diffusion Network (Emrick, 
1977) the Study of the League of Cooperating Schools (Goodlad, 1975) the 
study of successful technical assistance groups (Moore et al., 1977) and 
the study of OD programs in schools (Miles et al., 1978). The major ernphases 
of these studies have been upon: 

• illuninat inq the interactions between external agents and 
school innovators; 

• examining the impact of external agents on the school at 
vt^rious 3tages in the change process; 

• examining the organizational strjctur'e that are necessary 
to support linkage arrangements; 

• iden ifying mechanisms, including linkage roles, for over- 
coming barriers to cooperation between schools, school 
personnel and knowledge providing organizations;- and 

• identifying organizational or individual characteristics 
which promote the development of "temporary problem- 
solving Systems." 

St rat egiesK)f-change theorists also emphasized the importance of 
multiple stages in the problem-solving process. The following' four-stage 
model, where two of th*: stages are divided into substages, is a consistent 
representation of most conceptual frameworks (see Havelock, 1969; Rosenblum 
and Louis, 1981). Ine four stages are: 

• Problem identification 

• Problem refinement 

- consideration of alternative so3:tions 

- solution selecti^' 

• Imp lement at ion ^ 

^ planning for implementation 

- implementation 

• Incorporation 



20 27 



These stages of the problem-solving process imply a rational, linear model. 
However, these stages are also ponsistent with nonlinear proulem-solving 
approaches, which often characterize school change efforts (Sieber, Louis and 
Metzger, 1^72). An example -^f this nonlmearity is shown in Figure 2-1. 
Note that the figure does not imply that all probltJm-solving processes 
begin with problem identification. For example, d school may enter a 
problem-solving process by attempting to implement a new program which has 
been selected — not because it is intended to solve a problem, but because it 
has been mandated by the superintendent, or because it represents a new 
approach that the principal learned about in a graduate course. If the 
attempt at implementation reveals a poor match between the school's needs 
and the innovation, this may, in fact, motivate the school staff to begin to 
analyze what the real needs are for new programs. 

'j 

'!t rategies-of-change researchers emphasize a specific set of change 
outcomes. This approach tends to identify organizational health or gener- 
ally improved organizational functioning as a desired end state. Thus, for 
example, outcomes that may be measured include the quality of the change 
process itself, the alleviation of the problem that stimulated the attempt to 
change, or the improvement of the school's ability to become adaptive through 
the incorporation of improved jroblem-solving practices . 

An examination of recent reviews of tne strategies-of-change litera- 
ture (Louis and Sieber, 1979; Paul, 1977; Lieberman, 1977) reveals several 
theoretical deficiencies in this approach. 

• A true synthesis, as envisioned by Havelock, has failed to 
emeroe. Each study tends to generate its own hypotheses 
or variables de novo , and there is frequently little 
attempt to relate findings to a more general model of 
change strategies. 

m As pointed out by Berman and McLaughlin (1975), Yin et al. 
(1976), and Grbss et al. (1971), primary attention has 
generally been given to the relationship between the 
external agent and the client, and the initial adoption of 
an innovation. Less attention has been paid to the later 
stages in the change process, including implementation and 
mcorporat ion.* 

• There is a lack of conceptual commonality evident when 
examining the roles of external change agents. A variety 
of theoretical typologies of linking roles exists within 
the literature. Ey omples include the work of Sieber 
(1972)i Havelock (1969), Corwin (1977), Miles (i?77), 
Butl^ and Paisley (1^78), and Griffin and Lieberman 
(1974). There is, however, no generally accepted set of 
dimensions regularly found across these various models, 



♦Note that even the NDN evaluation (Emrick, 1977) which developed 
ari elaborate set of measures of implementetion does not analyze the outcomes 
of the progrc'^ 'n great detail. Rather, the emphasis of the study is upon 
describing program tactics that are associated with implementation. 
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FIGURE 2-1 
Th« Probtom-Sohrlno Proettt 
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even though some variables form the basis for several 
thepries. In addition, despite the critical" nature of the 
role, there is a lack of empirical knowledge available to 
help policy makers with the task of organizing the roles 
and improving the-^f Feet iveness of the individuals who 
staff them (Lieberman, 1977). * 

• Little systematic attention is paid to the user context , 
as opposed to the strategies of intervention. Since the 
strategies tend to be associated with particular change 
efforts (the Pilot State Dissemination Project, the 
National Diffusion Network, the League of Cooperative 
Schools), the lack of systematic attention to organiza- 
tional variables at the client level limits the usefulness 
of the studies for generalizing to a bropder model of 
organizational change. 

• Critiques also include a broader conclern that, on the 
whole, the writers concerned with the <hanaq6ment of 
change have not adequately attended to theoretical warn- 
ings about the relatively chaotic environmeht in which the 
process of change occurs. These include the "garbage can" 
or "organized anarchy" perspectives (March and-Olsen, 
1976; Sproull et al., 1978) and the "evolutionary" per- 
spective (Farrar, OeSanctis and Cohen, 1980). These 
theories of orgai;iizational chaos have considerable rele- 
vance for the development of a better understanding of how 

^ and when change can be managed in particular settings. 

In summary, despite the promise of the early 1970s, only 1 imited p rogre ss has 
been made in developing the "strategies of change oersi yect — V\fhile useful 
case studies continue to be produced, increased opportunil'res are, needed for 
comparative studxes of intervention in schools. 

Innovative Organizations/Natu. 3l Systems Perspective 

In contrast' to th^^"strategies of change" studies, the "innovative 
organizations" approach limits its inquiry to the study of change which is 
initiated m the absence of a collaborative relationship with external 
organizations or individuals, such as the one characterized by the RDU 
program. In f^ct, some studies falling into this perspective deemphasizp the 
importance of external stimuli other than the availability of" funds. The 
major focus of inquiry among this growing group of studies is upon th^ 
importance of the user context and upon finding correlates of change. 

Among the major variable groups that are often included in such 
stuciies are: 

• structural features of the organization, including size, 
complexity, formalization, structuring of decision making 
(centralization), and "loose coupling"; 
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• characteriatics of the achool's ''iechnology/' such as 
the degree of Individualization^ curriculum focusy or 
special staffing arrangements such as team teaching; 

a organizational climate , Including morale of staff members, 
administrative support for Innovation,- levels of conflict, 
and past lnnovatl.veness; 

a characteristics of organizational personnel, ^uch as' 

training experience , and professionalism; • 

a charact^erlstlcs of students, or other x:llents, such as 
racial composition, or achievement levels; and 

a characteristics of the organizational envlronn^nt, such as 
region, wealth, or political context. 

Major recent studies in this tradltlo^ Include: the Rand Change 
Agent Study (Berman and McLaughlin, 1977); the study of Organizational 
Influences on Educational innovations (DeaF, Meyer and Scott, 1975); the 
Mudy of adoption of innovations (Baldrldge and Burnham, 1975); the Teacher 
Corps Study (Cofwln, 1973); the Rural Experimental Schoola Evaluation (Rosen- 
blum and Louis, 19B1^; and the Daft and Becker (1979) study of innovation in 
high schools. 

The innovative organizations pet^pective has contributed in a major 
way to our understanding of ^innovation, largely by identifying features of 
the organization which help to explain why some organizations appear to be 
always on the cutting edge of innovation, while others constantly fail to 
respond to new ideas or procedures that are developed ^rom cumulative knowl- 
edge about leanVing and edgcational processes. In addition, because of the 
use of large-sample aurvey data, such studies have t^een able to cut through 
some of the conventional wisdom**' surrounding organizational change, while 
confirming other principles that have been developed by earlier case study 
approaches. For example, early siDciological research contended that large 
bureaucracies were inherently nonadaptivj* and resistent to change; innovative 
organizations research has shown, however, that larger and more complex 
organizations appear to be mcgre adaptive than smaller ones, largely due to 
the existence bf slack resources to support change efforts, and to their 
greater decentralization of decision making. Perhaps the major contribution 
has been to identify some of the features of the organization and its envi- 
^ ronment which condition rchange, and which make a purely "rational" model 
.of change inappropriate (Berman and McLaughlin, 1975; Downs and Mohr^ 1976; 
Eveland et al., 1977). ^ rj 

In particular, the resebrch into t^his general category has 
explicitly pointed to the fact that the context in which change ih being 
introduced will have extremely powerful Impacts upon the outcomes of the 
innovative process. As Greenwoodet al. (1975) pointed out,, researchers 
interested in educational change must "go beyond the detaila of the innova- 
tive project, and incorporate characteriatica of their aetting — the complex 
organization." 

•# 
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The fifxlin^ that ^ structural features of the school and its context 
will affect the outcomes of change programs is an extremely important one, 
for it points to the limits of most externally funded change programs. Many 
school characteristics, such as dize, complexity of staffing arrangements and 
staff characteristics, and climste factors, such as general morale, are 
relatively difficult to alter pven under the most ideal circumstances and the 
richest array of resources. Community contextual variables, or studefit 
characteristic variables, are virtually nonmanipulablei^ The expectation that 
a citegorical grant program, or even a more intensive intervention utilizing 
external expei>tise, might change these characterirstics in directions which 
may .better support change is unrealistic. These variables might, for ex- 
ample, help to explain the 75S attrition rate that oc^curr^d between the 
awareness and adoption stages of the NDN program (Emrick, 1977). 

' Another recent contribution of the innovative organizations appfoadn 
has been to suggest, features of organizational design that rmjst be taken }J^o 
consideration when change strategies are developed. Emerging from the 
innovative organizations approach ^is the notion tf.at the effectiveness of 
different change strategies themselves might vary between organizations with 
different types of structures. (See Firestone, 1980.) For example, the 
concept of "loose coupling" (Deal, Meyer and Scott, 1975; Weick, 1976; 
Rosenblum and Louis, 1981) refers to the notion that the 'linkages between 
educatignal subsystems are relatively weak compared with other types of 
organizations, such as thitse in industry. Deal et al. (1975) have noted that 
many schools tend to be doubly segmented: classrooms within schools are 
relatively autonomous and frequently schools within districts are relatively 
autonomous. The degree to which there is "tight" versus "loose" coupling may 
have a critical impacl on the degree to which change strategies designed at 
upper administrative levels will actually be implemented at lower levels. 
This,, in turn, suggests that different change strategies may be needed in 
different types of sichools or -districts. 

A major strengtli of the innovative organizations approach is that 
it tends to stress as outcomes the critical question of whether there is any 
objective evidence that change has taken place in the school. Until recent 
ly, emphasis was plared primarily on the adoption of new programs^ but 
recent studies have begun to examine the degree tc which programs are 
actually implemented as planned (8erman et al., 197A; Yin et al., 1976; 
Fullan and Pomfret, 1977). As many have pointed out, perceived implementa- 
tion of an education program may not reflect real changes in the school 
(Charters anc} Pellegrin, 1973). 

Studies have also begun to examine the persistence of programs over 
time, or incorporat ion . Incorporation represents the stage where new 
programs or processes are so completely established in the- school that they 
are no longer innovations. Recent evidence from the Rand Study (Bermah et' 
al., 1974; McLaughlin, 1976) and the Rural Experimental Schools project 
(Rosenblum and Louis, 1981) (both of these were studies of programs where 
outside' funding was d major stimulus to innovation) suggests that implemented 
programs are fragile, and are frequently dropped or diverted even before the 
end of the funding period. While federal funds are clearly a stimulus to 
participation, the level of funding is not significantly related to either 
program effectiveness or i'ncorporation (Berman and McLaughlin, 1977). Since 
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M Mtpect that Mat prt3graiii changaa in achoola do not receive significant 
axtarnal finaricial aupporti there ia reteon to pureue further the study 
of factora that effect incorporation. 

Oeapite ita rftrengthai the innovative organizations perapective has 
recently received aa^ie vary aerioua criticiam from^ previoua aupportera. 
Aaong the deficiencies cited are; 



a lack of attention to proceaaea of change , which may con- 
tribute to levels of implementation; 

a lack of attention to the characteristics of the innova- 
tion, particularly the organization's perceptiona of the 
characteristics of the innovation; 

a undifferentiated conceptualization of the organization! 
which doea not take into account variations within organi- 
zationa (aome type^ of decisions may be centralized, some 
nay be decentralized) or changes oyer time; and 

a a tendency to rely on croad-aectional surveys , which 

inherently limit the ability of the atudiea to addreas ^ 
iaauea auch aa those mentioned above. 

In addition I i^ is important to note other weaknesses of the approach when it 
ia viewed in the context of policy reaearch; 

i 

e In general, there ia no attention to the types of vari* 
ablea thrt have been fruitfully examined in the strategies 
of change reaearchi or in the older traditions of dissem* 
ination/dif fudion reaearch » auch as social interaction or 
organizational choice. 

e In their dpaire to find con&tant predictora of organiza- 
tional changoi there haa been a tendency to deemphasize 
policy manipulable variables or managerial questions in 
favor of looking at atatic predictors. 

e Concluaidna and generalizationa are often phrased at a 
lev3l of generality thai ia tob vague to be helpful (e.g., 
'*A atrong principal role is important."}* 

As Oowna and Hohr (1976) point out, however, one of the major prob-* 
leme of thia approach ia the fact that the researchers have not been able to 
achieve their goal, they have not been able to find a limited set of vari* 
ablea that are aasociated with change acroaa different organizations in 
different atudiea. Rather, we have found that similar variables, aimilarly 
neaaured, have diffe^nt relet ionahips with implementation when different 
aamplea are used. 
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INTEGRATING THE TWO APPROACHES: , A MODEL FOR THE STUDY OF THE RDU PROGRAM ' 

In the pre\rious sectj ion we idenu i f ied two major t rends .in the 
study of educational change. Out discussion was' pot meant ' to imply that 
" the two perspectives are jnutually exclusive, or that none of the studies cited 
above used variables or addressed issues from both 'groups. In large measure, 
this IS a matter of emphasis. Thus, fior ex^ple, we classify the Rand Change 
/ ' Agejpt Study .(Berman and McLaughJin, 1975 and 1977) as falling into the 
irSnovative organizations perspective largely duevto its more .limited emphasis 
on external influence variables, and to its relatively limited conceptualiza- 
tiori of change process variables.' The NDN evaluation* (Emrick, 1977), on the 
other hand, did measure organizational characteristics of the schools, as 
well as tactics, but did not emphasize them in the analysis. Other studies, 
such as Corwin's (1972) research on t+ie Teacher Corps or Daft and flecker 's 
(1978) study of ^^jph sphooZ innovations are. more balanced, in ihclu(^ing and 
analyzing elements of both perspectives. 

Two point\ that have emerged from ^our di^ussion should be- etnpha-^ 
sized. First, both the "innovative organizations" and the "strategies of* 
* change" approaches have certain theoretical weaknesses as well as certain r 
• important strengths. Second, the findings of several important studies of 
recent federal change efforts indicate that each perspectij^ can contribute 
in major ways to our understanding of change and innovation i^n education. 

It is also important to eniiphasize that our understanding of educa-"" 
tional change, although considerably more sophisticated now than it was five 
years ago, is still- quite limited^ . Mann (1976) summarized the dilemma by 
stating: ^ * 

> 

Programs were planned, curriculum was developed, teaching/ 
learning units were packaged, teachers were trained, and the 
results were 'frustrating, ^uneven, unexpected and temporary. 
What happens inside the schopl, at the service delivery 
level, is absolutely related to our success or, failure, yet 
the gap in our knowledge about implementing change in the 
schools is formidable, (p. 313) 

In order to begin the task of developing a better understanding of» 
the process needed for the dissemination of knowledge to improve local 
problem solving, it is essential to synthe&ize existing streams^of research. 
One of the ways in which this can be ^accomplished is to incorporate in this 
research sets of variables emerging from the two traditions that have been 
shown to be powerful in their ability to explain the complex phenomena under 
study.. Sucft an integration should draw upon both the explanatory or indepen- 
dent variables tnat have been utilized as, well es upon the differing defini- 
tions of what constitutes appropriate outcome measures of success within 
change programs. 

In addition to drawing upon variables that have already been defined, 
a synthesis should also attend to some of the conceptual weaknesses in each 
approach that have been described above and that are not fully addressed by 
other approaches. The development of a model to guide 'the study of the R&D 
Utilization program provided an opportunity both to complement and synthesize 
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the knowledge acquired from other efforts in dissemination and educational 
change. This opportunity emerges both from the nature of the program 
and the design of the research based on the program. On the one hand, the 
tt&D Utilization program incorporates in its structure elements of s strate- 
gies of change approach including: 

• the field agent function; * 

• tht: packaging and dissemination of .information; 

• 4*$^ provi'sion of technical assistance in problem identifi- 

cation and solution selection; 

• the provision of technical assistance in the implementa-- 
tion of a solution; 

• explicit attention to the stages of change; and 

. • active involvement of the client system in the process. 

On the other hand, as is strongly advocated by the innovative organi- 
zations/natural systems approach, a study of the RitD Utilization program must 
also take ifito account the importance of local conditions and problems . 
Thus, the study must consider the importance of the nonmanipulable and 
manipulable school context, the strategies and tactics of change, and a 
complex operationalization of outcomes. Although this study cannot deal with 
all the potential variables pertinent to the change process, it can bring 
together many of the issues and thereby, complement what has been learned from 
other program efforts. 

Put simply, we believe that it is important to strive toward a set of 
models of educational change which take into consideration some of the major 
contingencies that appear to condition planned innovation. However, given 
the complexity of the educational system, and the range of innovative activi- 
ties that schools and school systems engage in, it seems fruitless to con- 
tinue with the. hope that additional research projects will produce a single 
model thai is capable of psrsimoniously explaining the innovation process and 
its outcomes, and is also so-f f iciently detailed io provide insight to 
managers of change, whether principals, superintendents, or field agents. 
Rather, it seems morp likely that' progress will be made by developing contin- 
gency af^rdaches that can identify distinctive and different patterns of 
innovative behavior that may occur in different circumstances, and for 
different outcomes. » " ' 

A general schema which has guided the study and analysis of the 
impacts of the RDU program is displayed in Figure 2-2. This model is divided 
into three sections: 

• local conditions , which include concepts and variables 
derived largely from the "innovative organizations" tradi- 
tion and which are based on the assumption that local 
characteristics and the problem context condition and 
interact with external interventions and local decision- 
making behavior; 
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FIGURE 2-2 

A SCHEMA FOR EXAMINING THE IMPACT OF KNOWLEDGE UTILIZATION ON LOCAL SCHOOLS* 
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• intervention strategies, which map the RDU approach onto^ 
the "atrategies of change" tradition, and which cate^rize 
the strategiea employed by RDU into three different types 
--information, technical assiatance and an internal 
problem-solving process;, snd 

e proximate and diatal outcomes, both those intended by NIE 
and/6r the projects and those unintended. 

I 

This ^iji<i)le schema was derived not only from a synthesis of the 
research literature and concepts noted above, the design of the RDU program, 
but also from initial observations in the RDU schools and projects. Early 
viaitd to the seven RDU projfect headquarters and a sample of nine schools in 
the RDU program enabled ua to understand more completely the nature of the 
intervention (in particular^ that there were three components including the 
internal process activities). 'We were thus able to derive insights .^out 
the interactions between the components of the intervention (i.e., the 
direction of the arrows in the model), and to diacover that intended program 
in^acts (i.e., adoption, implementation, and incorporation of a new program 
or practice*, plus engagement in and incorporation of a problem-solving 
process) were accompanied by unplanned spin-off effects, such as staff 
development and organizational changes. We now turn to a brief description 
of the elements in the schema and the relationships between them, beginn^-ng 
with the intervention strategies. 

The Intervention < 

The RDU strategies for cbange included three project design features, 
each of which was presumed to have a direct impact on school improvement 
outcomes. The first two of these were most heavily influenced by the parti- 
cular design of each of the seven projects. First, each project developed 
.and administered a knowledge base consisting of RAD products (largely curri- 
culum innovations) supported by some form -qf validation or other evidence of 
impact. These products varied from highly prescriptive curriculum packages 
for which there were training and support' materials available, to simple film 
strips. Characteristics of adopted products (such as their qualirty, diffi- 
.'culty of implementation, need for local materials development, etc.) y^ve 
hypothesized to affect school outcomes. . « 

Second, two kinds of external human assistance were provided to 
schools through most of the RDU projects: a field agent ("linking agent," 
"faci^tator" or other generalist) who was employed by th^ project to support 
the schools in. their activities for the entire problem-solvxng period; and 
episodic training which was typically intended to assist the schools in 
implementing their chosen externally developed product, or in supplementing 
-it with materials as necessary. (The role of the* external agent corresponds 
to the role advocated by the. linkage model of the strategies-of-change 
perspective, in which sOch agents were recommended for theip potentj|al 
contribution ^o improved problem solving.) The nature, scope and intensity 
of external technical assistance were hypothesized to directly affect both 
the internal problem-solving activities and the school improvement, outcomes. 
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A third component of the intervention was the requirement of partici- 
pation by local school personnel m a variety of problem-solving activities 
through the various stages of the change process. This feature corresponds 
to the assumption of strategies-of-change researchers that involvement of the 
client system is a crucial factor in the success of a 'change effort. All of 
the projects ' attempted to provide structure and criteria for this process, 
although they had less dir^ct influence on the internal processes than 
on the external products that were made available or on the external human 
assistance intervention. The process was, however, an important feature of 
the RDU approach and several features of the process (including the level of 
tffort, quality of the problem-solving process, and breadth of involvement) 
are hypothesized bo directly affect both the choice of externally developed 
products and the school improvement outcomes. 

Local Conditions 

' Jn keeping with the assijnp'tions underlying the innovative organiza- 
tions/natural systems approach to the study of educational change (i.e., 
local site characteristics condition the school improvement process as well 
as the outcomes of a change effort), the model contains two categories of 
local conditions as independent variables. These are the relatively nonmani- 
pulable site characteristics , such as community characteristics, size of 
school and district, and types of staff and pupils, as well as the somewhat 
more ' manipulable structure and climate of the schools. The model also 
contains a category called problem characteristics , i.e., the characteristics 
of the problems that were being dealt with under the aegis of the RDU pro- 
gram. Local conditions are hypothesized to Condition the internal problem- 
solving activities, and to have a direct impact on school improvement 
outcomes. Much of the literature on educational change asserts that features 
of ^e local context (including student body characteristics) are the 
major predictors of the success , (or lack of success) of an intervention 
strategy, and in fact far outweigh the power of any intervention to affect 
school change. A major objective of the analysis in this volume is to 
test this assumption and to determine the relative impact of local conditions 
vis a vis the power of the RDU intervention strategies. 

Outcomes 

^ 

The final section of the model deals with the intended and unantici- 
pated potential outctomes. This section has been divided into two categories. 
First, proximate outcomas include client satisfaction with the services deli- 
vered and with the' process, and scope of implementation of the selected 
products. These, however, are not ^i^iewed as the ultimate potential impacts 
of^-4he intervention.* Ultimate, more distal impacts have be^n defined 
m t^rms of the intended changes — cont inued use and incorporation of the 
implemented products, incorporation of the problem-solving proi^ess, and 
^esol^^ion df the problem which was treated' by the RDU intervention--as well 
in terms of other impacts which were found to be spin-off effects of the 
problem-eolving effort. These include staff development outcomes and organi-^ 
zational changes. 



Our schema for examining the impact of knowledge utilization activi- 
ties on local schools may appear to be unidirectional or linear, with local 
conditions on one end, and outcomes on thd? other- It is important to point 
out that the schema is merely a heuristic device — an attenrpt to visually 
simplify the pattern of relationships that has guided the inquiry and analy- 
sis. However, same local conditions and the internal problem-solving activi- 
ties ,ar expected to be affected by the features of the intervention, as 
well 38 expected to be predictors of outcomes. Indeed, many c*" the outcome 
variables are really ctianges in the conditions of local school features. 
Thus, for example, the intended pLitcome of "process incorporation" implies 
changes in the internal problpm-sol vi^ig activities in the school. The 
outcome of "problem resolutiJh' is anotfer way of saying that the "prob- 
lem characteristics" (portrayed as independent variables) have been affected 
by the intervention. Furthermore, the outcome of "organization3l change" 
implies impacts on the more manipulable local site conditions, such as 
morale,' the ways m which decisions are made^ and roles and structure;, within 
the schools. Since these outcomes represent an altered state within thr 
local site context, they have been given labels under the outcome category, 
rather than being portrayed m the schema by reversible arrows between the 
first two groups in the schema. 

Ovprview of the Remainder of this Volume 

The remainder of this volume explicates the analysis and findings 
based on the examination of relationships portrayed in the model (Figure 
2-2). Chapter 3 describes the methodologies used in the conduct of the 
study. Chapter 4 describes the outcomes of the programs and their measures, 
the interrelat lonstiip of the outcomes, and the different categories of 
"success" in the schools which participated m the ROW program. 

« 

The next three chapters present the impacts of the three aspects of 
the mterve )t ion: Chapter 5 describes tjie analysis of the impact of the 
product intervention; Chapter 6 presep^ the impact of external technical 
assistance; and Chapter 7 presents the/impact of the internal problem-solving 
activities. Chapter 8 describes the inripact of nonprogram 'features, and 
looWs dt the degree to which local site characteristics are related to the 
outcome measures. 

Chapter 9 explores previously unexplicated relationships within the 
model, such as the impact of site characteristicr on the internal problem- 
solving act ivit ie3, the relationship between external assistance ^nd the 
internal processes, and the relationship between the internal activities and 
the characteristics of the chosen products. Chapter 9 also provides a 
sumiTiation of the model, including analyses of such key questions as the 
comparison of the pnwer of th^ separate intervention strategies vs. the power 
of the combined int ervsnt ion strategies, and the relative impact of the 
intervention vs. the power of local site characteristics to predict outcomes. 

Chapter 10 describes the costs of the RDU approach* at the 'ocal site 
level, and the relationship of these costs to the outcome measures. Chapter 
11 concludes the volume with a summary of the findings and their implications 
for educational change strategies. 
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APPROACHLS TO INTEGRATING QUALITATIVE AND UUANTITATIVE METHODS 

IN THE RDU STUDY 

>^ 

INTRODUCTION / 

The methods that were used to develop a data base to study the 
impacts of RDU at the school level were complex, and in many ways relatively 
unconventional. The purpose of this chapter is to not only describe the 
various data sources that were available to us, but also to place our method- 
ological approach in a broader context of debates about how bes\ to conduct 
policy and evaluation research. In particular, the chapter will emphasize 
our persistent attempt over the course of studying disspmination, k.iowledge 
utilization and school improvement in the RDU program, to take advantage of 
some of the strengths of both traditional qualitative and quantitative 
methods. 

METHOOaOGICAL DEBATES 

For the past 30 years, it has been common to refer to the exis- 
tence of two distinctive "paradigms" governing the methods of social scierice, 
enquiry. The first paradigm stresses the need to apply research design end 
analysis principles derived from the "hard" sciences, and emphasizes the 
dP'^irability of experimental or quasi-experimental design and statistical 
analysis. A second paradigm argues that social phenomena are essentially 
different from those observed by the hard sciences and that, in order to 
understand them adequately, we must understand the ways in which they occur 
naturally, and their meaning to members of the social structure. A "holis- 
tic" understanding of human aocial structures and behaviors requires a 
qualitative, observationally based methodology rather than experiment j1 
manipulation and analysis of a select number of variables. As r^^cently as 
1977, one observer of these two camps commented that the gulf between them 
was so great that it was unrealistic to assume that there would be any "grand 
synthesis" in the foreseeable future, and that any steps toward synthesis 
were on the "fringes" of paradigms (Rist, j977). 

However, there are a number of ir Hications that a need for something 
more than simple detente between the cam^s is growing. Some experimental 
method© logists, for example, have recently taken tentative steps toward 
acknowledging not only the existence of an alternative paradigm, but also its 
suit^ility for studying phenomenon which have typically been dominated by 
quantitative approaches (Campbell, 1974; Cook and Cook, 1977). Similarly, 
researchers who are advocates of qualitative methods have also called for 
greater attention to standardization of analysis procedures (Sieber, n.dO. 
Finally, a number of key articles and books have advocated integration 
between qualitative and quantitative methods within the same study (Sieber, 
1975; Lazarsfeld, 1976; Cook and Reichardt, 1979). The expressed need for 
integration is not occurring only at the fringes of social science disci- 
plines, but is also supported at the center, and is becoming more wide- 
spread. The tidiness of the divisions between camps is clearly breaking 
down: one can no longer assume that an anthropologist is totally ignorant of 
statistics, and traditional experimental psychologists are hotly discussing 
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the problems of small N designs for which preferred statistical techniques 
are inappropriate (Herson and Barlow, 1976; Krachtowill; J977). 

The movement toward integration between qualitative and quantita- 
tive methods has been fostered most evidently in social policy research for 
several reasons. First,* some of the afirly high aspirations for quantita- 
tive social policy research were debated by an accumulation of "null" 
findings, and "black box" research designs were unable to reveal why appar- 
ently massive experimental treatments should produce no measurable effects. 
Thus, committed empiricists began to look at qualitative research methods as 
an approach that might help them to improve their analyses — either to lead 
them to interaction effects that should be explored, or to allow them to^ 
account for otherwise inexplicable findings, or to help them in other ways.* 
Other researchers, such as Gross et al. (1971) or Charters and Pellegrin ' 
(1972), raised the question of whether or not a treatment had actually been 
implemented as part of an elaborate experimental or qu^si-experimental 
design. These studies also inofecated the difficuKy of determining the 
degree of implementation without some qualitative understanding of what 
constituted implementation for a given program. 

Second, the most rigorous and sophisticated of designs has not 
eliminated doubts about the durability of policy research findings. Rather 
than eliminating controversy over the results of social policy and evaluation 
research, rigorous designs have simply added new questions for debate. 
Any observer who does not like the results of a major policy study can almost 
invariably finft^a variety of methodological or analytic flaws which can be 
claimed to undermine its validity. Not surprisingly, some polic> makers have 
arrived at a deep-seated skepticism about the durability of supposedly "hard" 
findings — at least where they are unsupported by qualitative data which make 
sense in the light of ordinary knowledge and experience (Corbett and Fire- 
stone, 1980; Sundquist, 1978 ).♦♦ 

Third, there -are also practical considerations which have promoted 
the use of qualitative methods. The increased need for "forms clearance" 
procedures that are required before standardized data collection instruments 
can be used under federal contract regulations should not be underestimated 
as a burden, both for federal agencies and researchers. Since forms clear- 
ance can take from four to . six months, the federal agency that asks for 
qualitative data m addition to quantitative data can begin to "knov/' some- 
thing about the topic in question long before a survey or testing 'program 
begins. Thus, particularly in cases where there is only limited interest in 
a "bottom line" assessment, qualitative approaches may be perceived as more 
efficient. Qualitative designs may also be viewed as more flexible in 

♦See Sieber (1975) for an extensive discussion of the ways in which 
qualitative research can be used to complemef^t^a design which is predomi- 
nantly quantitative. ) 

♦♦That policy makers have come to vi/ew "soft" approaches as fruitful 
is evidenced by *he significant increase the nurrber of RFPs from a variety 
of afgencies which require qualitative* Vr case-based approaches rather^ than 
(or in addition to) quantitative Ones. ^ 

♦♦♦That field-based methods are now viewed as efficient is an ironic 
turnabout from earlier periods when "public opinion" surveys and other survey 
data collection activities were ^touted because of their speed and and low 
cost. 
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responding to changing policy contexts and questions than traditional 
experimental designs. 

It should be emphasized that thp pressures cited above do not simply 
describe a shift in emphasis from one camp to another; rather they illustrate 
a desire on the part of policy makers (and at least some researchers) to draw 
upon the best of both methods. Despite the increase in policy makers' 
support of qualitative research, there* continues to be limited interest in 
sponsoring true, ethnographic case studies, except in the context of supple- 
menting v^ry larger well defined social experiments (see, for example. Trend, 
1976; Herriott, 1980). The new emphasis upon qualitative methods does not 
seek a paradigm shift; rather it retains the strengths of quantitative 
research««generalizeability of results, reliability of observations, and the 
ability to synthesize a large, complex study in a brief report. Increas- 
ingly, there has also been a strbn^perceived need to address the integration 
of- findings across different methodological approaches in a more formal 
way. 

THE STdOY or RDU: METHODOLOGICAL APPROACHES 

The m'sjor distinctive characteristics of the methodological approach 
used in this study are: 

• the merging of qualitative and quantitative data within as 
well as across sites; 

• ^af f 1' V, patterns which involve senior researchers who 
participate ' i both quantitative and qualitative aspects 
of the study; 

• persistent attempts to triangulate data sources and inter- 
pretations; and 

• cyclical interaction between the qualitative and quanti- 
tative method during nil phases of the study, including 
sampling, instrumentation, data collection, analysis and 

~ reporting. 

EacI- of these features of the approach, as they were developed in the study 
of tie R&D U*:ilization program, will be discussed briefly below. 

Program and Research Context 

Like many demonstration efforts, there were delays in funding the 
research component of the RDU program. NIE was, however, particularly 
anxious not to lose data on the early development of the program, and there- 
fore funded a regional laboratory to perform some data collection during the 
, first months of the program. NIE also encouraged each of the seven demon- 
stration projects to design relatively elaborate data collection systems, and 
funded them to hire researchers to write a total of 42 case studies about 
particular sites, which were site specific and relatively unstandardized. 

Abt Associates prooosed in 1977 to supplement the existing data 
collection activities with two waves of survey data collection that would 
tap cross-site issues during early and later stajges of iiTiplementation. The 
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agfency, however, expressed a preference for conduc^ing a single survey, and 
asked the research staff to consider alternative ways of providing longittj- 
difial analysis of the prdcess of change in schools. With NIE support the 
research staff developed a design that, while initially considered far from 
ideal, finally produced a data base that allowed effective integration of 
qualitative and quantitative data across approximately 90 local sites 
involved in t.he program. 

f 

In the study of the R&D Utilization program the emergent interaptive 
design was in large measure .a by-product of external features of the study 
context. • At the time, many of these constraints were perceived as albatiob- 
ses by the research staff who, when itiitiating the study, had a marked 
preference for the sequential model described above. However, in the process 
of coping with external constraints, problems with availability of data and 
data quality, a great deal was learned about how to maximize the utility of 
different approaches, "found" data, and' both ad hoc and systematic informa- 
tion. In other words, the design that emerged is robust against many of the 
"normal crises" which occur in field-based policy research. The specific 
elements of the design are discussed J)elow. 

K ey Data Collection Strategies 

The study of ROU impacts at the school level used three major data 
collection strategies: ^a_SMI>*^ of teachers apd principals, unstructured 
case studies, and "standardized case studies." In addition, several sup- 
plementary da*^a sources are used in analyzing program effects at the site 
level. Each of these ?ipprqach6s are described in greater detail below. 

Survey of Teachers and Principals . A mailed survey of teachers and 
principals in schools that had not officially "dropped out" of the program 
was conducted in the Fall of 1979, after the end of the operating program of 
the seven RDU projects.* In five of the projects (Pennsyl vania, 'the National 
Education Association, the NETWORK/ Consortium, the Northwest Reading Consor- 
tium and Florida), the survey wps sent to the universe of principals whose 
schools had been in the project for at ledst two years. In Michigan, only 18 
of the. 51 districts involved in that state were eligible for the survey. 
These sites had been sampled prior to our participation in fhe study as a 
manageable number frun which to collect data in this relatively large pro- 
ject, which involved 49 schools.** 

In both Michigan and Georgia, where in rt.any cases the entire district 
was a target for intervention, a single school within each district was 
sampled. In each case, the school recomi.iendat ion was made by asking a 
knowledgeable participant (a project employee or a district employee in some 
cases) to identify a school that *had actually received services under 



*Approximately 1055 of the schools that initially agreed to partici- 
pate dropped out. Dropouts occurred so early in participation, however, that 
it was deemed inappropriate to send these ex-participants a questionnaire 
about events that occurred two years previously. 

**The sampling criteria for the 18 were purposive rather than random, 
and attempted to reflect the variety of different problems and types of 
sites in the project. Sampling was carried out by the High/Scope Educa- 
tional Foundation in Ypsilanti, Michigan. 
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the program and was implement inq (or had considered implementing) a school- 
based product. In both Michigan and Georgia, th^re is a known sample bips 
toward schools that were somewhat more involved in the program. This was 
considered appropriate because it matched the strategy of these two projects 
of allowing the district to determine vhich schools should receive servicer 
under RDU. Thus, the schools in these two projects were more comparable to 
those of the other five projects, in which pre-target ing of service delivery 
.by -the project was-4»orfe typical. 

Survey administration was carried out by AAI^s Survey Research 
Group. The. survey form was mailed to 199 principals, and returned by 152 
after twd follow-up letters and a phone call. This produced a return rate of 
76S.* In the case of teachers, each project assisted us in generating a list 
of all the teachers in the sampled schools who were eligible to use the 
product that was adopted (or being considered for adoption). The potential 
user group varied from the entire faculty in most elementary schools to the 
members of a single department in some larger high schools. In each in- 
stance, we sampled the universe of "potential users" if l^he number in the 
school was less than five. Where it was greater than five-, we rancjomly 
sampled five teachers to whom questionnaires were sent. Procedures for 
questionnaire administration included two follow-up letters, but did not 
include telephone calls. 1246 questionnaires were sent out, and 594 returned 
with a final return rate of 48%. A detailed breakdown of return rates by 
project ,is presented in Table 3-1. The table indicates that return rates 
were lowest in those projects (CEDISS/Michigan, Georgia, and NEA) where 
teacher and principal participation was known to be lowest. Follow-up phone 
calls affirmed a bias favoring respondents' who were more actively involved" in 
implementation. 

Copie*^ of the questionnaire instruments are presented in Appendix D. 

Case Studies . Ao noted above, part of the pre-existing design for 
the study, which was developed before our participation, was a set of 42 
"site case studies" which were'written under contract to the seven ROU 
projects. In most cases, the projects delegated the responsibility for 
preparing case studies to another individual or set of individual^. In 
Michigan, for example, the case studies were prepared by staff members of the 
High/Scope Foundation. In Florida, all six case studies were prepared by a 
professor at a state university. 

In a few cases, a project staff member — the project evaluator — wrote 
one or more case studies* In the i^iorthwest Reading Consortium case studies 
involved collaboration between project staff members and staff consultants 
from the Center for ^Fducat ional Policy and Management at the University of 
Oregon. AAIhad no direct respons it i 1 i t y for form ing the case st ud ies. 
However, some indirect influenre nrnirred «s « result nf two rnnfprenpp*? for 
^ase study writers that we, conducted to discuss issues of common concern. 



♦The feasibility of conducting telephone interviews was explored.^ 
However, it was clear that principals wto had not ^returned the questionnaire 
had either left the school (11 cases) or were not sufficiently knowledgeable 
about the program to con)plete the questionnaire. 
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Table 3-1 

RESPONSE RATES FOR TEACHER AND PRINCIPAL SURVEY 



PRINCIPALS TEACHERS 



Project 


Sent Out 


Retjurned 


% 


Sent Out 


Returned 


me 

% 


Northwest' Reading Consortium 


26 


19 


66% 


185 


92 


50% 


Georgia 

/ 

Pennsylvania ^ 


28 


17 


61« 


li'y 

• 


55 


46% 


17 


17 


100% 


. 112 


68 


61% 


Consort iun/NETWORK ^ 


23 


' 21 


91% 


205 . 


102 


50% 


NEA . ' 

• 


25 


17 


68% 


172 


67 


39% 


Florida 


26 


23 


38% 


262 


129 


49% 


CEDISS/Michigan 


52 


38 


73% 


191 • 


81 


42% 




199. 


152 


76% 


1246 


594 


48% 



I 
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Wa also developed a list of issues of interest to us, which we asked Hie case 
study writers to use as an index. In general,' however, our advice about the 
content and focus of the case studies was neither sought nor supplied. 

The use of case study writers hired and paid by the projects inevi- 
tably raises a question of objectivity, both in selection of sites and in 
analysis and reporting. In mbst cases site sampling was carrieid out within 
the first six months of the RDU program's operations, so that selecting for 
success would appear to havB been impossible. In most of the projects, 
however, random sanjpling procadures were not used, and so the possibility of' 
a preference for sites that "smelled like success" cannot be ruled out.* In 
terms of objectivity in data collection and analysis, the RDU project staff 
members were deeply respectful of and ^committed to the research component of 
the program. Ih selecting external contractors or consultants, care was 
taken to choose competent professionals with no ties to the schools that they 
would be studying. Discussions with individual case study writers l^jad us to 
believe that they felt free to write about their sites as they saw them. In 
addition, the project staff members who were responsible for preparing or 
supervising the preparation of case studies were also open about negative 
findings emerging from the cases, as well as positive ones.* Interestingly, 
case studies written by staff members of the seven RDU projects tended to be, 
if anything, more critical in their intepretutions of events and outcomes at 
the school level than subcontractors or consultants, perhaps because they 
were more familiar with the ideal models that each project had developed. In 
sum, we do not perceive that there was any srystematic bias favoring data that 
would point to positive outcomes in the schools that were covered in case 
studies written under project supervision. 

The quality , of the case studies as sources of data, on. the other 
hand, was highly variable. The length of cases ranged from approximately 40 
pages to nearly 100.. While length was certainly associated with the level of 
detail of the data presented, the degree to which the case contained a 
convincing, documented, "holistic" portrayal of the school's progress through 
a problem-solving process was more a function of the skills and experience of 
the case study writer. 

A final major source of data was field notes and site reports based 
on visits to 51 RDU schools. Nine of these site visits involved one day 
orientation visits by two AAI staff members very soon after Ihe beginning 
of the research project. An additional 42 schools were visited for two 
days by two AAI staff members during the* Fall of 1978 and the winter of 
1979. Finally, a sample of 21 of these schools were revisited for a two day, 
one person visit in the Fall of 1979. 

The procedure for sampling the 42 schools visited by us involved 
asking each project to nominate the site which they believed best exemplified 
what they were trying to achieve, the one which they Relieved to be least 
promising, and a random sample of the remaining sites that were not involved 
in a project-sponsored case study. The 21 follow-up visits were selected 



*In one project, an independent contractor eliminated from the selec- 
tion process sites -which the ste*^f believed would be unlikely to become 
involved in project activities at all. This was the only instance of a clear 
bias away from potential failures. 
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based on the degree to which staff members* as a group felt that they might 
illufflinlite problems and' issues in carrying out the ROU process, and thus 
included both sites that appeared to be highly successful, and some which 
appeared to be chaotic and of low potential. At lea^t one site with low 
potential for success and one with high potential was selected in each of. 
the seven projects^ 

Our approach to conducting these case studies was to draw upon 
the emerging standardized case method . The main features of the method, 
ab described by Yin (1980 and 1981), are: 

• considerable pre-specif icat ion of the data that are to be 
collected in extensive protocols, which include not only 
questions to be answered, but specification of documents 
to be collected and at least some of the respondents who 
must be interviewed or observed; 

• built in flexibrli^ to pursue at least some additional 
topics and interviW different individuals from those who 
are pre-specif ied if they appear to be locally important 
to the social phenomenon under study; 

• an emphasis upon a u nit of analysis that is larger than a 
respondent , e.g.,' a .social unit, a classroom, a program 
as it is implemented in a local context, etc.; 

• an emphasis upon early data reduction and analysis while 
in the field, and the requirement of a standardized 
reporting format wKich involves pre- i dent ifying at Ifeast 
the majority of issues to be addressed; 

• the use of brief, but iterative approaches to data collec- 
tion . Typically there will be an initial round of field 
visits, a period of analysis and refinement based on these 
and a second round to collect data that are missing be- 
cause of changes in the design and analysis plan, or to 
observe changes over time (most studies using this ap- 
proach spend between one and two weeks at a given site); 
and 

• the development of causal arguments within each case and 
across cases using a direct replication design. The logic 
of this analytic approach is to find specific phenomena 

in repeated cases under predictable conditions. Where 
the N is very small (five or less) the approach will 
typically emphasize the internal validity of causal anal- 
ysis within each case; where somewhat larger, the design 
often looks for systematic replications and attempts to 
establish the variability of conditions under which a 
phenomenon occurs. 

Some flexibility was delegated to each site visit team in deciding 
the specific method of writing up a narrative report of site visit observa- 
tions, although each report was required to address specified topics. "he 
case ma' erials were nob, however, written even as rough ""cases."' Rather, 
they w' re more on the order of organized field notes. A typical field report 



from a site visit would range from 15-25 typewritten pages, excluding docu- 
mentary materials. The intent of each field report^was to provid e primary 
data for future analyses rather than final conclusions. Site visitors were 
instructed to limit their "hunches" or interpretations to specified sections 
of their reports. The development of causal arguments occurred in analysis 
seminars, which were held after completing each round of field visits, and 
during which the site visit staff would attempt to develop and defend hypo- 
theses about cross-case patterf;s using various site visit materials as 
evidence. These preliminary causal arguments were later expanded into many 
of the "hunches" underlying the analysis presented in this volume, and into 
additional case materials and analysis presented elsewhere (Louis, Kell, 
Chabotar and Sieber, 1981). 

• 

One observation that is critical to understanding the standardized 
case method is the importance of iterative site visits; In many of the 21 
schools that were revisited there were important and sometimes dramatic 
changes over the elapsed jtime. Some schools that looked like failures had 
blossomed into successes. Others had undergone a reverse process. In still 
others,, there were many "pritical events" which altered the participants' 
reactions to or feelings about the program and their involvement, and which 
changed our estimates of the likelihood of more permanent impacts at the 
school level. In sum, a single visit cannot, in our opinion, provide valid 
data about a complex organizational process that unfolds over a long period 
of time. 

% 

Other Data Sources at the Site Level ; In addition to the major data 
sources discussed above, there were a variety of additional data sources, 
some of which were designed and collected by the Abt Associates^ research 
staff, and some of which were intenaed primarily to serve other purposes. 
Among these were: 

• data on the field agencies who served each of the sites. 
These data were collected primarily throqflh surveys of 
active RDU field agents, and were intended to contribute 
to an analysis of field agent roles in educational set-, 
tings. However, we also intended to examine yie rela- 
tionship between agent role definitions and site per- 
formance. (The methodology used for obtaining data 
from field agents is more tully described in Louis and 
Kell, 1981); 

• data about the design and cervices that characterized 
each of the seven projects' strategies for effecting 
change in schools. These data are presented most ex- 
tensively in Louis and Rosenblum (1981), but were also 
intended to be used to determine whether project de- 
GiyM afwj iiiciiiciijcijH:)itL dTfticLeu bite uulcomes; 

• various documents and surveys that were collected as a 
part of the preliminary study carried out by 4:he first 
evaluation contractor, including demographic data about . 
each site, and a survey 6f "key informants" collected at 
the beginning of the program. 
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• "event-'triqqered" reports , discussing different * 
phases of the school's progress through the pro- 
jectt which w^re msnagement reports to NIE for 
monitoring purposes; 

e a case study writer's survey , which obtained som 
standardized data on almost all of the case study 
sites. 

e various other documents , provided either by the 
projects, the site, or other individuals. 

MERGING QUALITATIVE AND QUANTITATIVE DATA SOURCES 

The study was designed so that each data source could provide dif ferm- 
ent types of information. Thus, the principal questionnaire emphasized 
information about the characteristics of the school, and about institutional- 
ization of the process snd product. The teacher questionnaire emphasized 
teacher assessments of materials, the process and the impacts of the activ- 
ities and process on the school. The site visits, on the other hand, 
emphasized capturing the nature of the intervention at the school level, 
particularly the major features of the problem-solving process at various 
stages, and the level of activity of various key actors, both inside the 
school (principals, teachers, etc.) and outsic^e (field cgents and others). 
There was some overlap in items and topics between instruments, but the 
strategy was to develop an information "division of labor" based on the 
knowledgeabi^ity of th,e respo, ^ent/observer about certain topics, and 
the need to develop both site-viSj.t protocols and questionnaires that were 
not overly burdensome. 

Merging Qualitative Data Sources Within Sites : 
The /Consolidated Coding Form (CCF ) 

Much of the discussion of integrating qualitative and quantitative 
methods essentially involves cross-site !3nalytic issues. The challenge 
facing us included the cross-site merger of different data sources, and even 
more pressing was the problem of a diversity of within-site data sources. 
One feat'jre of policy and applied" research is that it is frequently very 
"messy." For example, most major programs are not studied simply by one 
group of "evaluators" or researchers. Rather, there may be internal evalua- 
tions, reporting and administrative data collection from the funding agency, 
and a sequence of external researchers. The multiplicity of research and 
reporting . requirerents is a source of major concern to those who are heing 
studied and who may frequently complain that "somecMie was here just last 
week asking me the sp*^ question." This was a particular problem in the RDU 
program, where "mini ethnographers," NIE administrative reporting require- 
ments, two sets of external evaluatorb, uikJ u vdriely of uLiit^i lebeaicii&ib 
interested in the program all descended with regularlity upon the schools 
involved. This is increasingly a dilemma in other programs as well, such as 
Title I, IVc,' etc. . 

For any site, our information could include any or all of the data 
sources mentioned in the previous section. However, in no more than 20% of 
the sites was a complete data base available, and in^ most cases major 
instruments or documents were unavailable. In sum, we had a "missing data" 
\ problem of the first magnitude. For example, the generous assumption that 

o « , 
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either a case study or a site report could count as "case" data yielded an Kl 
of 90 usable "case" sites. (For only 75 of these sites did we have both a 
teacher and a principal questionnaire.) However, if we looked for specific 
data—e.g., the level of effort devoted to each phase of the problem-solving 
process— many of the project-produced case studies had vague information, 
while most of our own site reports were quite specific. Not surprising- 
ly, given the qualitative, narrative nature of much of the data available to 
us, non-comparability within d given instrument or data category was often 
equally severe. Thus, in some instances the "event-triggered" management 
reports to NIE read li.ke 5-6 page mini-case studies, and were filled with 
very specific data about what occurred at the site during a particular 
phase. Jn other cases, a report on the same^topic (sometimes evgh from the 
same project) might be Yio more than a page long, and contain little or no 
useful information. Yet, despite these problems, it seemed foolish to throw 
away "evidence" of any sort, particularly in light of the high- level of 
effort that had gone into collecting some of the data that were missing for 
the largest number of sites. | 

Rather than analyzing each data^3ource separately,' it was decided 
to combine all data sources for each site in an "intensive" sample (e.g., 
the 90 sites where we had a usable case study, or which had befen visited by 
a project staff member). This was done by developing a consolidated coding 
form in which a single score or rating was given for each variable on the 
basis of a senior researcher's judgment derived from all the possible data 
sources discussed above (with the exception of the formal principal and 
teacher surveys'). The measures were developed to maximize the completeness 
of available data. Thus, for example, the standardized case materials 
stressed precise measures of level of effort devoted to planning and imple- 
mentation in terms of staff days. The case studies tended to have much 
less precise measures. Level of effort was, therefore, coded as "high- 
medium-low," where these were dgfined as ranges of staff days. Other sample 
items may be seen in Figure 3 
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Responding to these site-based questionnaires (which included 240 
dichotomous or Likert scale items) was an extremely time-consuming task, and 
involved between two and three days 6f reading materials and verifying 
responses on the coding form. All coding was conducted by core senior staff 
mennbers, who had made visits to at least four of the sites, and who went 
through an intensive two-day session in which common interpretations of 
items were emphasized.* 

While the process of "sleuthing" through the data trail for each 
case did consume a great deal of time, what resulted was a "quantitative" 
data base for 90 sites, which covered issues that could not be easily tapped 
through traditional survey methods. These included the "quality" of the 
decision-making process and patterns of influence of different actors over 
JeuioiuMo. Decauac of the divcrcity 2nd Isrge ntjmh^r nf qUpqj morp tradi- 
tional forms of data reduction and analysis would not have produced a data 
base which reflected both the "holistic" knowledge that site visit teams 
brought to the cases and the reliability of standardized data, integrated 

*A copy of~ the coding form and coding instructions is available 
m Appendix D. * 
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Figure 3-1 
SAiJEL€-rROM CONSOLIDATED CODING FORM 



103. in your opinio^, tu what extent »MS^the faculty as a whole actively^ involved in the 
jiiOitwtfitition process**i.e. • to what extent did the facuUy as a wnole purlicipate In 
dU'u$sibfis, making decisions, or carrying out tasks related to the process? (CIRCIE ONE) 

To little or 00 extent 01 12 

To SQM extent 02 

yro a great extent ^ • 03 

To a very great extent 04 

w Hissing dau -1 

Conflicting data -2 

Not applicable -j 

104. In your opinion, how much Influence did the faculty as a whole have over the major decisions 
In tKo Implementation prokress? (CIRCLE ONE) 

None or vtry llttl^ 01 ^ 12 

Some ' 02 

A great deal 03 

ttlssing data -1 

Conflicting daU . . . . , • . . .\ , . -2 

Not applicable -3 



CARD 12 
<cont.) 

24-25.26 



27^28,29 



ICS. ^Ourln^ the Implementation stage, was there a formally constituted group— other than the 
faculty as a whole— specifically empowered to make de.c1s1ons or carry out tasks related to 
implceienutlon? (CIRCLE ONE) ^ 

INPQRTANt: The group should meet the following criteria: 

• It must have a label (although this may be informel). 

• It must Include at least two district or school staff. 

• It must include at least one "potential implementor.'* 

Yes ^ 01 

Mo : 00 

Hissing data -1 

Conflicting data . . -2 

Mot applicable : -3^ 

SKIP TO QUESTION 111 



1 2 



30-3U32 



106. How many jnembers of the group you described in Section IX were also members of this group? 
(CIRCLE ONE) 

None 01 

Pew" (less than 20!:) 02 

Some (20-495). 03 

• A large proportion (50-79S) , 04 

All or most, (over 805) OS 

Hissing data -1 

Conflicting data -2 

Mot applicable 7 -3 
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33-34*35 



both within and across sites.* And, while time-consuming, this method was 
considerably less costly than other altecnatives^(such as ignoring non-con- 
forming data sources or gathering additional data on site in orcter to 
develop more complete standardized case studies). 

Staffing 

^ The two cultures of research often result in research designs 

where staff members are specialists in either qualitative data collection 
and analysis, or quantitative data collection and analysis. This e^sproach, 
while appropriate for the parallel model , does not f acilit ate integration 
and interaction. The staffing patterns in this study involved having the 
same core staff of senior researchers involved in all instrumentation, data 
collection and analysis. Thus, every individual who contributed as a major 
author or analyst to the study was personally familiar with site processes * 
at all levels, and had responsibility for some portion of the quantitative 
analysis in the study. Conversely, all site data collection wastarried out 
by individuals with a major role in the analysis. This "integration by 
staffing" is perhaps one of the most effective ways of ensuring that the 
cycle of testing quantitative and qualitative observations occurs on a 
regular basis. 

Triangulation and Data Quality 

As Webb et al. (1963) have pointed out, qualitative data become more 
ijompelling if observations are supported. by multiple sources of evidence or 
, observations. The issue of reliability was of deep concern to us, in part 
because of the rather varied nature of our underlying data sources. Thus, 
our design involved several approaches to triangulation: 

• Inter-observer . Site visits were conducted in teams. 
Teams were required to reach consensus in preliminary 
rating of sites on quantitative dimensions. In 
addition, whrii using the consolidated coding form, 
inter-rater leliabilities were conducted. 

• Holistic vs. categorical . As part of the interweaving 
of qualitative and quantitative data, findings- from 
the qualitative data bast were repeatedly tested out . 
in the qiantitative and vxce versa. For example, in 
attempting to develop a categorical variable summarizing 
"site success," a definition of success was first dis- 
cussed arpong the staff on the basis of the field data 
visits.** An indicator that reflected these discussions 
was developed from the sucvey data of principals and 
teachers. The ^0 "int^rtsive" sites Vieve then categor- 
ized, using the quantitative indicator, and staff ^gain 
discussed whether they were correctly classified, by 

' «75 limited inter-rater reliability check was conducted which .re- 
vealed an agreement rate of 72%. Several sites were eliminated because of 
the amount of missing data. 

**This variable is described in greater detail in Chapter 4. 
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uainq the "holistic" judgements that they developed in 
the ^ield visits, or by reading the "mini ethnographies." 

• Cssft vs. survey . In severs! cases, similsr messures were 
built into teacher snd principal surveys, and into the 
consolidsted case coding instruments. Correlations be- 
tween the perceptions of local respondents, end the per- 
cef;tions of case raters were calculated, and several var- 
iables were discarded where correlations were not posi- 
tive and significant.* 

• SiTvey va« survey . In all cases we were trying to obtain 
building-level measures of the process and outcomes of 

) mpleffienting new curriculum and staff development mate- 
rials. This meant aggregating teacher responses to the 
building l^vel. In order to ensure that we were not fall- 
ing into the "ecological fallacy," an analysis of variance 
between and within buildings, using botli teacher and* prin- 
cipal data, was perforrned, which indicated that, for most 
measures, varidbility between schools was higher than 
variability within schools. 

• After the latter two activities were completed, measures 
were scaled to form a single school-level score, reflect- 
ing the responses of -principals, teachers and the * 
external case material collected by field staff. 

Cyclical Interaction Between Quantitative and Qualitative Data 

In many multi-method policy research designs, it is assurried chat 
qualitative data will influence quantitative data collection and analysis, 
but lot vice versa (Sieber, 1975). Most discussions of integrating quan- 
titative and qualitative methods also promote integi jtion at one phase of the 
execution of the study (Smith and^ Louis, forthcoming). Thus, fox example, 
the sequential approach promotes integration during the period of instrumen- 
tation and final design for the "final" quantitative data collection, the 
parallel approach promotes integration only after al 1 data collection and 
analysis in the separate streams is completer^, and the standardized case 
approach promotes integration during the process of cross-case analysis. At 
thi^ time counts and frequencies, (or even correlations) are made. 

Lt\ \he RDU study, on the other hand, quantified data were available 
from a very early point in the study, both because of the surveys that were 
*co,»ducted prior to the award of the contract to Abt Associates, and be- 
cause data from the first round of field visits were coded while we were 
on site. The results of early descriptive analysis changed the focus of 



♦For a further discussion of case-survey tr langulation, see Appendix 

B. 
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our qualitative data collection in signif-cant ways, since they highlighted 
A-opics where there was little variation between sites (and certain issues 
that were expected to dominate the qualitative data collection were therefore 
eliminated). In addition, they suggested some issues to look for — such as 
the relationship of racial diversity to the process of change — that were not 
initially picked upby the field visit or case study writers as important. 
This cyclical interaction between qualitative and quantitative data occurred 
throughout ^;ie study, during staff analysis seminars, and during the construc- 
tion of instruments at various phases of the study. Cyclical integration 
stresses the following features: 

• Interaction between qualitative and quantitative dala 
in sampl ing . The interactive approach al lows purposive 
sampl ing for cases to be combined with random sampling 
(or sampl ing of the universe) for survey or other 
structured data collection, This feature enhances 
discovery and general izeab i 1 it y . 

• Interaction between qualitative and quantitative data 
in instrumentation . A constant interaction between 
qualitative and quantitative analysis procedures produces 
an iterative approach to instrumentation, both for field 
data collection and more standardized coding instruments 
or data collection. Iterative instrumentation would be 
almost impossible for mc_t standard longitudinal research 
designs, which involve repeated measures. This problem 
IS avoided in the "consolidated coding" procedure, where 
repeated measures are reconstructed ^om a broad eviden- 

t lary base. 

% Interaction between qualitative and quant itat ive data in 
dat a col lect ion . The development of survey-type instru- 
ments for use even in early field data collection for- 
ced the research team to seriously think through 
measurement assumpt ions at a stage where, in a more 
typical sequential or parallel study, it would not 
occur. We believe that this contributed significantly 
to the quality of our measurement. 

• Interaction between qualitative and quantitative da t a i n 
analys IS . In the interactive model, it is impossible t o 
identify a distinct "analysis phase" in the research 
p.oject. Rather, analysis begins with thp first data 
collection p'vent, and occurs periodically throughout the 
project. More importanily, however, is the fact that the 
simultaneous analysis of quantitative and qualitative 
data by the same staff continuously requires a testing 
and verification of one data ^urce against another, in- 
creasing both reliability ana ,dlidity. 
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« Interaction between qualitative and quantitative data 
in report ing > While some of our reporta are more 
^'qualitative" and othera more "quantitative," the 
imnersion of all staff in both kinds of data has 
meant that no report draws solely upon one type of 
data source. ^ 



SOME REFLECTIONS ON THE POTENTIAL STRENGTHS AND WEAKNESSES 
OF THE INTERACTIVE MODEL FOR SOCIAL POLICY RESEARCH 

Since there has been only this one "test" of the approach that 
we have described above in a major educational study, any assessment of its 
viability must be preliminary. However, it is appropriate to draw some 
conclusions about what has been learned regarding" the integration of quanti- 
tative and qualitative methods in design, data collection and analysis. 

Design 

Since the beginning of the "evaluation" movement, many disciplinary 
researchers have complained about the lack of attention on the part of policy 
makers to designing "evaluable" programs. They further complain about policy \ 
makers* tendency to involve researchers too late, and at too' low a level of 
effort, to carry out the optx'nal research designs (Rodman and Kolodny, 1964; 
Weiss and Rein, 1969). Unfortunately, most of these pleas have not been 
attended to because of a variety -of reasons which are often beyond the 
control of the research branches of various government agencies. Thus, as 
policy ^ searchers, we' are typically forced to choose between a research 
context that is less than ideal, or a situation where we do not conduct 
research on interesting and significant policy endeavors. The "sleuthing" 
strategy that characterizes the interactive model is premised on the assump- 
tion that the least promising circumstances can yield usable and even 
exciting data^ that almost any piece of information can be turned into a 
"clue" to understanding the phenomenon in question, and that systematic 
analysis of .clue^is important* 

The in|era^ive approach should not necessarily be viewed as an 
ideal research design, for it lacks the elegance or simplicity of a true 
paradigm. However, ^t is particularly well suited to addressing some of the 
realities of policy research context which often cause stresses and strains 
in more elegant designs. Among these realities are: 

• where the study combines both significant exploratory 
and/or evaluative and hypothesis-testing components; 

a where it seems important to have both a "rich" or 

holistic understanding of a process occurring in a field 
setting, and a broader crosr-case analys^j which ad- 
dresses some of the same issues; 

a where the policy audience is composed both of people 
who prefer qualitative "valid" data, and quantitative 
"general izeable and reliable" data; and 
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• W^re the field"TeaIity is "messy^^—lots of previously 
collected data exist but there ia a great deal of var- . 
ianbe in quality aiid depth of information between sites; 
where response burden is a significant practical or 
political issue; where resources may be too limited to 
begin anew with a totally new design; or where the 
research involves documenting a longitudinal process 
that is already well under way. 



The interactive approach is not a simple one, however, and causes its 
own problems. It requires constant attention from staff members who are 
committed, consistent, in-house senior staff, and each of whom is capable of 
both qualitative and , quantitative data collection and analysis. One feature 
of the approach that should be emphasized is the need to maintain very low 
rates of turnover among project staff — since th? processes of design, data 
collection and analysis are intertwined, any need to replace a key person 
involves considerable costs in socialization. In fact, our attempts to 
replace key staff mernbers during the last year of the project were not at all 
successful; indeed it became necessary to extend our work over a longer 
period/of time in order to complete the analyses. In addition, it must be 
emphasized that staff members themselves must be relatively free of paradig- 
matic preferences. If either the project director or other key staff members 
are reluctant to become equally involved i-n qualitative and quantitative 
analysis, promoting cyclical integration may be extremely difficult. 

Data Collection 

The wide variety of different types of data collection activities 
that were carried out in the RDU study allows us to address issues regarding 
valldity/reliqt)ili ty that are often debated among those who are concerned 
with the increase in multi^-site qualitatively based studies. 

One of the major informal debates that occurs among qualitative 
cross-site researchers is the question of how one can best preserve the 
"holistic" insights obtained in a traditional ethnography with the necessary 
truncating of time in the field that results when financial resources are 
stretched to cover many sites. The proponents of the "standardized case 
study" tend to prefer brief on-site data collection activities which are 
conducted exclusively by in-house staff members (e.g., b staffing approach 
that is more similar to traditional quantitative data collection procedures 
than to the traditional ethnographic field work). The individuals who argue 
for "mini ethnographies," on the other hand, contend that even a superficial 
understanding of the functioning of a program, organization, or other social 
unit in a local setting must involve more time on site and more visits over 
a period of time, in addition to greater flexibility in obse.rvations (Knapp, 
1979). Typically a "mini ethnography" approach argues for hiring trained 
individuals— of ten consultants — who are geographically closer to the system 
under study, and who can thus use whatever time is available for observing 
and interviewing in a way which is more responsive to the unfolding of events 
at that site. 



The study provided an opportunity to assess the "richness" and 
depth (validity) of data collected using a "mini-ethnography" Approach 
(project case studies) and the standardized case approach (our site visits). 
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The ^ypic^i ^iini-'ethnogi^hy irf^ about 15-20 

days of data collection and very preliminary within-site analysis on the part 
of the case study writer who was, on average, a university professor who 
came to the job of writing the case studies because of personal interest.* 
As a result there was, in a number of cases, a great deal of "contributed" 
'time. , The typical standardized case study, on the other hand, consumed 
approximately foyr-five person days of data collection, which occurred at ^ 
single point in time for most sites, but with a return visit .for approxi- 
mately 35% of the sites. Two-person teams were used (except for the return 
visits which were conducted by a single person) and they conducted* some 
joint interviews with pre-ident if led key informants and some individual 
interviews with teachers and other relevant respondents. The standardized 
site reports were written by onB team member on the basis of a team debrief- 
ing and discussion; each case was reviewed by the other team member and 
augmented where necessary. > 

On the whole, we believe that our standardized case studies yielded 
data that were as -insightful into local site-speeif ic prqcesses as the more 
intensive mini-ethnographies. In addition, they were typically far more 
useful for the purposes of ' cross-site analysis. In a nunrtber of instances, 
we obtained both standardized cases and mini-ethnographie§ on the sarr ^ 
site.- Where there were differences in what was emi hasized in each case, 
in most instances our data were equally detailed, o id equally informative 
about the longitudinal processes of change. In no case where we were famil- 
iar with the mini-ethnography sites did we find any major contradictions 
between the data that we had collected, and the data that were collected 
through more intensive, less standardized means. 

Two conclusions may be drawn from this informal comparison, which 
• was corroborated .not only by our possibly biased project staff, but also 
by several individuals who served as consultants to the study who were 
familiar with both types of qualitative case data. First, the more costly 
mini -ethnography does not result in sufficiently better data to justify 
the additional use of Resources (although some [e.g., Wolcott, 1981] would 
argue that the process of truncating and focusing field data collection and 
analysis may violate the most basic definitions of an "etfmoqraphy" ) . Second, 
the standardized case study can reveal local site uniqueness (if the reporting 
formt allows this) and, in addition, is far more useful for cross-site syn- 
thesis. Mini-ethnographies are no better at reflecting "holistic" patterns 

♦The appropri ateness of the background of case study writers for 
conducting Jit his type of research varied considerably. In addition, some 
of che 47 "mini-ethnographies" actually .used a standardized case approach. 

♦*It should be emphasized that the 4-3 days per site that yielded 
relatively valid data for our study should not be generalized to other 
set.tings. For example, when examining district -wide iDehavior, it is clearly 
necessary to have a somewhat larger number of person days than when examining 
program functioning in a single school. However, the general principal of 
contrasting mini-ethiographies and standardized case materials still holds. 
Further, it should also be einphasized that informatijn gathered during the 
4-5 days on site was augmented by review of documents rTior to and- after the 
visit, and through familiarity with the nature and sub£,tance of the interven- 
tion gained in interviews with managers at the funded project level and with 
the external providers of technical assistance. 

50 




in a site because they still lack the essential feature of a traditional 

ethnography! an on-site pi esence oyer .,an_ex±end_ed period of time which 

involves substantial participant observation in addition to interviewing. 

Validity is not the only problem, however. Those who are concerned 
-about the growing use of qualitative cross-site analysis often point to the 
issue of relif(bility of field observations as a serious one. This concern 
may be even greater when one is attempting to . address very slippery and 
non-tangible fCBtures of organizational process such as "the influence of the 
principal over the decision-making process." *We believe that the deliberate 
attempts within the study to compare the data obtained from qualitative 
methods with more traditional survey methods indicat;e that, with care, this 
should not be regarded as qui*te as serious a problem as has often been 
thought. With adequate triangulation, ancj with a stable, sharing staff that 
is fully socialized in common definitions of the phenomena under study, it is 
possible to semi-structure the data collection techniques and still achieve 
high levels of agreement between different sources of data. 

Let us take, for example,- one of our less concrete measures of 
school-level outcomes, "overall organizational impacts on the school." After 
the first round of preliminary site visits, it was decided by the staff that 
there were many soin-off effects in organization development and improvement 
that were not, in all cases, directly tied to the implementation of a new 
curriculum package. These included changes in staff morale, improvement in 
the image of the school in the community, etc. We then developed a set of 
Likert scales reflecting side benefits that we had observed, and during oqr 
major qualitative data collection we rated all of the schools that were 
visited on this battery. The battery was also included in the surveys 
teachers and principals. While the_ field staff tended to rate the schools 
somewhat less generously than indigenous respondents pn broad school orgrni- 
zatiqnal impacts, the ratings that were given by the field staff correlated 
.55 with the princinal ratings, and .44 with the teachers. These are quite * 
' typical of the intercorrelations between data sources that were ach-ieved 
for items or indices which overlapped. ^ 

We were finally able to achieve even greater reliability in our 
quantitative data set by combining school scores on our outcorrie variableb 
both from originally qualitative sources (e.g., from the "consolidated coding 
form") and from survey data.* 



Data Base Construction Issues 

. , — < ^ 

The major .problem faced in constructing a data base from a variety of 
quantitative and qualitative materials is that of mis&ing data. Missing 
data, whether through survey non-response, or item non-response within a 
survey, are always a problem with complex data sets. Even in face-to-face* 
standardized data collection settings, item non-response is an issue where 
the questions being asked are complicated and difficult Vo answer. In 
^ general, where' multiple regression or other analytic procedures employing a 
large number of variables in an analysis are used, the item non-response 
problem is handled by simply limiting the dat^ set to items and/or respon- 
dents on which a relatively complete set of responses is available. Where 
multiple data sources are being used, where qualitative data are being 
transformed into quantitative data, and where the numbers of "respondents" 

♦For a more extensive discussion of the "quality" of the quantitative 
data that were derived from various sources, see Apf)endix B. 
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(m thi,8 case schools) is .quite small, the standard approach is simply not 
feasible. In our cBse, for example, among the 90 schools on which consoli- 
dated coding forms were available, six principals failed to respond because 
, they had actually left the school. When- this problem was compounded with 
item non-response, we quickly, realized that any regressions including more 
than ten variables might reduce the N for analysis from 90 to the loyi 40s. 

Our approach to solving this problem was two-fold. First, we decided 
that it was necessary to "retrieve" some crucial missing data in the CCF 
survey through estimation procedures (more detailed description of estimation 
procedures xs provided in Appendix B). Secon'cJ, we decided to let the remain- 
ing item non-response problems affect the Ns in our analyses as they would. 
Thus, in many chapters, the nunter'of cases included m a table may range 
from 179 (the N of , schools for which we obtained teacher surve]^^ data) to 55 
(in cases where we can combine data from principal, teacher and CCF sources 
but r\ which there is some item-le^el non-response). The degree to which 
this "wandering N" affect's the general izeability of our analyses between the 
small N tables and the universe of schools is discussed further in Appendix C. 

Integrating Qualitative and Quantitative Data in Analysis 

As has been implied above, one of the main features of the interactive 
approach is that this analysis does not occur after data collection is 
completed.^ Rather, analysis is an incremental feature associated with each 
'Site visit, between groups of field data collection activities, before the 
design of later instruments, and so forth. It was this ongoing process that 
permitted the research staff, as a group, to arrive at the consensus neces- 
sary to develop reliable, holistic cross-site coding schemes, which were the 
key features around which the integration of qualitative and quantitative 
data occurred. 

The process of trying to quantify our field data while these data 
were being collected was a key feature of the integration process. A field 
site team faced with the necessity of making joint assessments about the 
local site processes on a Likert scale, was also required to clarify their 
perceptions about the measurement properties of the scale, the phenomena that 
had been observed, the relative weight to give different re^spondents' per- 
spectives on the issue m question, and whether the most site-important 
features were being tapped by the coding instruments. This process, which 
occurred both m the field and during extensive debriefing sessions with the 
entire core research staff of eight, ensured that (1) there was a constant 
press to have quantitative instruments reflect, as much as possible, our more 
holistic understandi»^g of how sites were operating; (2) a more precise 
understanding of the site was developed through quantification; and (3) the 
existence of the quantitative forms 'Continuously pushed us toward the stand- 
ardization that was necessary in order to conduct a cross-site analysis of a 
very large number of sites. 

In the end, our analysis cannot be said to be either quantitative 
or qualitative as these terms are traditionally defined. For example, can a 
data base composed of numbers that are entirely dependent on the iterative, 
holistic judgements of experienced site field teansn be (^escribed as only 
quantitative? While the analysis procedures used to manipulate the data are 
statisticaXy the data, and any interpretations results, are totally 
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cond^i t ioned tyy th« r "artgrns^. On ttier~ather hand, as we approach any given 
analysis using case materials rather than quantified data, it has become 
genuinely impossible not to embed that activity in our knowledge of the 
descriptive statistics and* correlational relationships that were available to 
us well before qualitative data collection had ended. 

Many colleagues who have faced the problem of cross-site qualitative 
analysis with Ns of ten or more, have found it impossible to avoid some 
quantification of the data. The process of "holistic" analysis appears to 
break down at some point, and the analyst begins jotting down counts of 
occurrences of phenomena, possibly even computing rank order correlations, 
but unquestionably thinking as a quantitative analyst. At some point — 
perhaps where the N reaches about 15 — the ability of most people to hold the 
cross-case holistic story vanishes. There are two responses in analysis. The 
most typical one perhaps is to essentially throw away all but the uest or 
most familiar data informant's description. At this juncture,, some formal 
quantification of key variables may help. When, a? is increasingly the case 
in policy research, the number of sites is 25 or more and the number of 
informants may be in the hundreds, we are \inquest ion ably in a situation where 
formalization of data-analysis procedures — e.g., some- form of quantification, 
either more or less highly rigorous--is essential simply to manage the 
data. 



CHAPTER 4 

ThC IMPACTS, OF THE R4D UTILIZATION PROGRAM 



INTRODUCTION 

As our earlier discussion of this pp)gram suggests, the R4D Utiliza- 
tion program was ambitious in its 'aine. The program was not only intended to 
increase teachers' awareness and utilization of R4D products in local schools 
(product outcomes), biit also to have a more global consequence. It was hoped 
that the program would improve the ^ay schools identify and solve their 
problems, bbth in terms of increasing the breadth of participation in the 
problem-solving process, and by making the problem-solving activities them- 
selves more national (process outcomes^ .* ' ~ 

Early^ in the study of the ROU program, a series of nine relatively 
brief familiarization visits to schools participating in the seven opera- 
tional projects was conducted. The *aim of the visits was to talk with staff 
at these schools to become more familiar with how the program operated at the 
local level. Through unstructured interviews we learned that, in general, 
awareness and utilization of R4D products were in fact increasing through the 
u^e of improved problem-solving practices* However, it quickly became 
apparent that other things were happening at these schools as a direct result 
of their participation in the program: the schools themselves were changing 
in a variety of ways (organizational outcomes), as were the school per^^onnel 
(personal outcomes). This led us to expand the rang? of school improvement 
impacts to be studied under our research design to include organizational 
development and personal growth outcomes. These outcomes were reported by 
school staff members to be at least as important as the intended R4D product 
and problem-solving process impacts. 

In this chapter we present a descriptive overview of the various 
intended and unanticipated outcomes. The first part of the chapter focuses 
on a very brief overview of some of the survey responses from teachers and 
principals on items that were intended to tap knowledge utilization and 
school improvement in the schools. Here we look at four different categories 
of outcofliies: implementation outcomes; outcomes relating to the use and 
incorporation of problem-solving behaviors sought by RDU; organization 
dcvelopfment outcomes for the school as a whole; and personal benefits derived 
by pc^i'ticipating staff members. In the second part of the chapter, we turn 
our attention to a rather different question. Instead of describing how the 
outcomes were perceived by the individuals involved, we aggregate our outcome 



*Key characteristics of the rational model are (1) thorough analysis 
jnd prioritization of school needs or problems before searching for school 
improvement strategies; (2) ,a search outside the local school system for 
assistance and information, particularly in the search for solutions to 
problems; (3) systematic examination of alternative solutions according 
to explicit criteria; and (4) a focus on solutions which have been field 
tested and empirically validated. 
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date to the school level, and develop a fnodel for explaining how the various 
school-level outcomes are related to one another. Finally, in order to more 
firmly ground our description of the nays in which ROU affected participating 
schools, we present a simple typological classification of school outcomes, 
and present some vignettes about actual schools. 

PROGRAM IMPACTS 

V For some time, a policy debate has raged over the value of educa- 
tional R4D products. While proponents point to the importance of developing 
a "knowledge base" of programs proven to be effective, relevant to and linked 
with* clear and enduring pupil impacts, detractors complain loudly of "old 
wine in new bottles," irrelevance and low levels of utilization. In addi- 
tion, numerous studies have shown how quickly educational innovations are 
discontinued once funding evaporates or key school staff* move on to other 
assignments (Bermdn and McLaughlin, 1978). Our research, howeveif, clearly 
indicates high levels, of teacher satisfaction with adoptjed R&D products, 
and as we see in the following chapter, the characteristics of the products 
themselves are strongly related to a number of program outcomes. 

There has also been some question about the appropriateness of 
attempting to do organizational development (OD) within school^ and school 
districts (Derr, 1976). Yet, our data indicate that in general the ROU 
program was successful in achieving broad-based participation in improved 
problem-solving practices. However, subsequent reuse of this approach is 
uncommon, and prospects for more enduring impacts on schools are not always 
bright. 

^ Unanticipated program consequences inclucjed noteworthy impacts 
on the schools as :>rganizat ions , ranging from reported improvements in 
curricula, materials and teaching methods, to pupil impacts and improvement 
in the schools* images m their communities. Similarly, individual teachers 
often reported growth in leadership skills, promotions and increased re- 
sources for colleagues. Of particular interest was a conspicuous scarcity 
of reports of negative consequences from participation in this program. 

We now consider each of these several categories of program outcomes 
in greater detail. The data were drawn from surveys of principals and 
teachers in participating schools, as well as from site visits conducted 
m 90 schools (51 by the AAI research staff and 42 by case study writers 
engaged by the seven operational projects), and an examination of other 
documentation. In the discussion which follows, we focus primarily on 
the individual teacher and principal survey data. In the case of the tea- 
cher survey, this means that we present an overview of ♦'.he data before 
scaling, and before aggregation to the schof^l as the unit of analysis. The 
initial presentation and analyses of scaled and aggregated data occur later 
in this chapter in our discussions of "A Model for Examining Impacts of the 
R4D Utilization Program" and "The Range of Site-Level Impacts of the ROU 
Program. " 
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Outcomes for R&D Products 



* A major objective of the R4D Utilization program was to install 
an appropriate R&D product in schools participating in the program. Thus, 
the degree to which schools identified, actopted and implemented a product 
relevant to the problem they sought to alleviate is a critical measure of the 
interinediate or proximal success of the program. Other intermediate product 
outcooies include various aspects of teacher satistoc'don with the products, 
the nurtbers of pupils and the percentage of their school days affefcted by 
implementation, and how difficult the product was to implement, including the 
need for adaptation. 

Approximately 100 different products or sets of curricular materials 
were, adopted by the participating schools. The most frequently adopted 
products were reading packages such as the Wisconsin Design for Reading, 
Exenplary Center for Reading Instruction (ECRI), Houghton-Mifflin basal 
Management System, and San Diego Right~to-Read. Career education packages 
which were most popular included Career Development Centered Ctirriculum, It 
Works, and AEL Career Decision Making Program.^ Popular mathematics programs 
included Brevard County LAMP and STAMM. In general, these programs ranged 
frdm lists of objectives for teachers to detailed management programs; some 
included a variety of materials for classroom use, such as slides or film- 
strips and tape cassettes, student work and record-keeping sheets, and 
associated texts. 

The characteristics of the products themselves varied along a nurriber 
of dimensions, including whether they were R&D- or practitioner-based. 
For example, some products were intended for use in only one classroom, 
while others were to be implemented throughout the schools. Some, such as 
San Diego Right-to-Read, consisted of sets of ideas from which adopting 
teachers were able to pick and choose, while others, such as ECRI, required 
• significant, highly structured changes from all teachers and were therefore 
more difficult to implement. 

Of particular interest here is the fact that the products and mate- 
rials adopted were more frequently practitioner- developed — i.e., NDN prod- 
ucts — than the more formally developed R&D-based materials such as those in 
the NIE catalog. This may be due to a conspicuous dearth of R&D products in 
some areas such as 'career oducatiorjf where interest only burgeoned in the 
early J970s and the time available for producing a variety of relevant, 
validated products m time for the RDU program (which began in 1976) was too 
short . ^ 

Other areas in which validated R&D products were scarce include 
school- or district-wide planning, inservice training and basic skills at the 
secondary level . In the latter case , the need for products — especially 
reading at the secondary level--was not recognized until after the RDU 
program was under way. Finally, some schools needed assistance with topics 
stemming from racial and ethnic integration and»the special needs of minority 
groups. The avail ibl3 pool of products for bilingual students was relatively 
sparse. 
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Practitioner-developed products also had a logistical advantage 
because they frequently had exp>erienced trainers who were funded through 
other federal programs such as NDN and who could provide pre-implementation 
assistance and follow-up services to adopting schools. As we will see in 
later analyses, availability of training in product use was strcngly related 
to several measures of program success. 

Among the schools that had reached the "product selection" stage by 
the time pf our final data collection, over 80% of the teachers responding to 
our survey indicated that they were using the product or had used it in the 
past. Another 5% had definite plans to begin use in the future. Fewer than 
20% of the users reported that the products ne-^ded afiaptation to a great or 
very great e,xtent. Product use was at a high level: yover 65% of those using 
the products reported they used them with al>i^of_t6eir students, and 855o of 
the users stated the product was regularly used at least once per week. 
Satisfaction with the adopted products was generally high, with over half of. 
the users reporting that the products were directly relevant to the, most 
pressing problem in their school, that they filled a need in the classroom 
and provided new ideas. Another 25-30% of the users reported these state- 
ments were at least true "to some extent." 

The users did not encounter serious problems with implementing 
the products they adopted. About 20% reported the products required change^ 
in teaching style, changes in classroom organization or management, or 
substantial additional record keeping. Only about 9% reported difficulties 
in implementing the program or materials tc a great or very great extent. 

A more long-term, or distal, product outcome is the extent to which 
it is incorporated into the everyday functioning of the classroom — i.e., 
the extent to which product utilization is "routinized" (Yin, 1979).* At 
this point data from two sources become relevant. In order for the product 
to be incorporated not only must the teachers indicate that they plan to 
continue using the program or materials in the future — perhaps wi^h modifica- 
tions — but building administrators must indicate that certain stc|jo necessary 
to ensure the continued possibility of use have been taken. Thus, although 
83% of the users reported they would continue to use the products, it is 
still necessary to consider such lof.g-term questions as whether the product 
has been incorporated into curriculum plans, what measures have been taken to 
ensure that new staff use the product, etc. 



*We choose the term incorporation very deliberately, to reflect 
our conviction that curriculum innovations are rarely (if ever) fully 
institutionalized in schools. While structural changes (such as. kinder- 
gartens or .a middle school) become fully institutionalized for long periods 
of time, a school's curriculum tends to be evolutionary, both within the 
school as a whole, and as it is applied in classrooms. Thus, while we find 
that a high percentage of teachers and principals are committed to continued 
use bf the R&D products they selected, we suspect that if the schools vlfere 
visited five years from now many of the current components of the curriculum 
would have been replaced with new models. The belief that institutionaliza- 
tion of curricular innovations is rare does not imply a value bias on our 
part either in favor of constant evolution or stability in curriculum. 
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Building principals of schools that adopted products were asked 
whether a variety of such events had already occ^urred or would definite- 
ly occur in the future. In over 70% of the schools, th£ products had been or 
would be incorporated into curriculum plans. About 50% reported that written 
guidelines for product use had already been developed, and another 11% 
reported this would definitejy occur. Almost 60% report(&d that new staff 
would receive training or orientation in the use or the products, and that 
training or injservice for current staff would be used to ensure continued 
product utilizat ion • Over 90% report ed that some or all of t he ir teachers 
would use the products to some extent, with 62% indicating that the products 
would be used guite evtensively. 

Outcomes for the Problem-Solving Process 

: ^ 

In addition to the emphasis on having jan R&D product installed 
at a participating site, a mj^«^7ocus of the R&D Utilization program was to 
increase a school * s • capacity to deal with its problems by providing staff 
with training and practice in group problem-solving processes. Though not 
explicit in any .RDU project, it is implicit that there are-two critip^l 
aspects to this goal of improving problem solving at the site level: "^one 
involves the use of a rational problem-solving model , and the other stresses 
the need for relatively broad-based part icipat ion in problem-solving activi- 
ties. In other words, any and all groups which will be affected by the 
decisions reached should be present on the problem-solving team. Thus, the 
extent to which the sites actually used a rational problem-solving model, and 
the extent to which there was broad participation in problem-solving act ivi- 
ties, become two important intermediate outcomes of participation in the RDU 
program. ^Note that both could vary for each site across stages of the local 
process.) 

4 

As the program operated at the site level, these two goals were 
generally met while the site went through the problem-solving process. In 
most cases, a field agent was availa 2 to quid^ the site^s activities, and 
in some cases economic sanctions cou.d be ap^^lied should tbe site not **t'oe 
the mark." But factors inherent the process itself militate against its 
reuse at a later time — for example some staff resented the complexity and 
t ime-ronsuming nature of the process. Our conversations with site staff ^Iso 
revealed that even when they felt they could go through the process again 
without the aid of a field agent, the release time provided by the RDU 
program was often a sine qua non of its success--otherwise teachers could 
not spend the often substantial amounts of time the problem-solving model 
required. We must also remember that, in general, improving their problem- 
solving practices was not the main reason sites got involved in this program. 
This impl les that a more distal process outcome was the extent to which 
the improved problem-solving pr act ices--or at least, some of them--were, 
likely to be used again in dealing with other problems. 

In terins of breadth of participation in the problem-solving process, 
our data su gest that there was generally good representation of groups who 
would ultimately be affected by the decisions made. This was true across all 
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stages of the process, ' aHhough shortcomings other than broad representation 
were evident. In the case ^of 90 sites for which we had highly detailed data, 
we* rated them on their problem-solving and groLip decision-making activities 
in terms'-of a listing we developed of desired traits specific to each jDhase 
of the process.* Where a site's rating was reduced, weindicated specific 
types of deviations from these traits. These fel^ into interesting 'patterns 
across the various stages. 

At 92% of the sites, problem-solving teams were established, and 
there was generally good representation 'on th^'e teams of the groups which 
would be affected by the t,eams' decisiong. In addition, both interes]^ and 
attendance were high in most sites, and few members dropped out. However, 
during the early stages of problem solving (problem identification and 
solution selection), we found that decisions were.* often made or heavily 
influenced -by administrators or other external parties. This was true during 
problem identification at 36% of the sites, and djurio'^ .solution selection at 
24S of the .sites. During the later "^stages of problem solving (planning for 
innplementat ion and implementation), the continuity of formal decision-making 
groups'^was not upheld. This was true at almost 20^ of the sites. Meetings 
became leqs regular at 76% of the sites, and decision making at 21% of the 
sites did not involve all affected grpups during planning for implementation. 

In terms of the rationality of the process, we fcfind that many sites 
appear to have adhered closely to principles of sound problem solving in many 
ways. In the problem-identification stagjfe, for example, 80% of the sites 
appear to have carried out proolem-identification procedure^ as planned, and 
arrived at a problem definition that appeared acceptable to almost all of 
those that would be affected by it. During solution selection, 80% or more 
of the sites appeared to have selected a new and relevant solution that was 
acceptable both to potent ial implementers and adThinistrator s , and this 
selection occurred with a level of effort that seemed to us to be appropriate 
for the complexity of the task. Implementation, on the other hand,, was 
rational among the BD% or more of the sites that did not appear to have made 
inappropriate adaptations^, that obtained reasonable administrative support, 
that implemented at the approximate scope that was planned, and ,that relied 
appropriately upon external technical assistance and training opportunities. 

However, well over 40% of the sites shSwed at least one, and some- 
times several, departures from oui; ideal criteria. During problem-identi- 
fication activities, -the most frequent variant was that the problem defini- 
tion was merely a restatement of someone*^ a prion assumptions or pet theory 
(46% of the cases). Alternative definitions frequently were not posed ^nd 
considered .(43% of the cases),, and the problem was not adequately specified 
prior to beginning the search for solutions (34% of the cases). 



* A de t ai led d i scuss ion of the criteria that were used to document 
rationality of the process is included in Appendix F. Sample criteria are: 
the problem definition is clear, manageable and relevant to the situation; 
alternative solutions are posed and carefully examined; the solution is 
relevant to the defined problem; any adaptations of products are appropriate 
and* carefully thought through, etc. 
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During solution se||^ction, the most common deviations were that 
alternative solutions were not carefully examined according to a set of 
explicit criteria and evidence of a solution's effectiveness or 

suitability was not obtained (32%) • During planning for impleinentat ion, 
of the sites did not make formal plans for some or most aspects of 
implementation. 

During the implementation stage, adherence to sound practice was 
generally much closer, with only about a quarter of the sites showing any 
deviations. Most common among these were not taking ader^uate measures to 
ensure implen»entat ion of essential features and goals of the produrts (23?o), 
and adaptations of the products implemented* when this may not have been 
necessary (23%). 

The more distal process outcome measure is the extent to which 
the sites repeat some or all of the problem-solving piocess to solve other 
school problems in the future* Our data showed that 41% of the principals 
and about 34% of the teachers at participating schools said they had repeated 
(or were repeating) all or part of the RDU approach to solve another problem 
in their school. These parts of the approach included use of teams of 
teachers and administrators to make decisions, enlisting, the services of ah 
external field agent, using procedures for deciding among alternative solu- 
tions, etc. 

Outcomes for Participating Schools a s Organizations 

We have seen that the R4D Utilization program's objective of getting 
R&D products installed at part icipating sites was, in large measure, achie- 
ved. To a somewhat lesser extent, the program's goal of improving local 
problem solving was also achieved, at least for this one time. However, 
neither of these categories of outcomes necessarily means thit there will ber* 
any enduring changes in the schools as organizations. ' That is, the simple 
fGct that a certain set of activities was accomplished, culminating in the 
adoption and implementation of, for instance, a new reading program, does not 
mean that the school's curriculum was inproved, or that the new materials 
were in any way better than those used previously. Similarly, the organi- 
zational structure of the school, which is difficult to change under any 
circumstances, can survive other changes without alteration. 

However, as ' we quic/ly learned during our preliminary site visits, 
a number of unanticipated effects were occurring in the schools themselves 
and among their staff members. The spontaneous reports of such effects by 
teachers and principals in unstructured interviews led us to develop specific 
lines of enquiry into these organizational effects. When we asked teachers 
to serve as internal observers of what was taking place in their schools 
"'a methodolorjy previously used with great success in Abt Associates* eval J- 
at ion of the Rural Schools Program), 30-70% of them gave evidence of a 
variety of positive effects on their schools: improved curriculum; better 



♦Later analyses sh-^wed that local adaptation of the products was 
negatively related to proqr am outcome measures, ^e Chapter 5. 
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materials available; greater coUegiality among staff; and generally better 
teaching. About 40S of the teachers reported an improvement m school 
organization and management, improved decision-making and problem-solving 
procedures, and improved morale. About 45% of the teachers said the image 
of their school m the community had been improved. 

It IS true that 30-50!S of the teachers reported "no change" on any 
one of these dimensions, but only a tiny minority, generally fewer than 2% 
of the respondents, said these dimensions had been even slightly affected 
adversely. Compaiable data from principals of participating schools and from 
our research teams* visits to the schools confirmed these reports.* 

Outcomes for Participating Staff 

As a result of their participation m the RDU program, the staff 
of the schools involved had a variety of expi^riences: some received train- 
ing m ^roup proBlem-solving techniques; others had the opportunity to visit 
other schools or educational product developers to observe R&D products in 
use. Some staff received training m the use of an adopted product and 
returned to their schools ,to trairl their cdileagues; still others became 
spokespersons who visited other schools to tell of their own experiences 
using an R&D product. 

An anonymous questionnaire was used to ask participating teachers 
about the extent to which they personally benefited from involvement in 
the RDU program m a variety of' ways. In general, 15'-30% of the teachers 
reported they had benefited m the following ways to a great extent or very 
great extent: thtir teaching and leadershio skills had improved; they had 
learned about curriculum development; they had morv^ self-confidence and 
new resources for helping their colleagues. Another 30-40% reported these 
benefits "to some extent." Inc-tteased self-confidence and^ job satisfaction 
were also reported by many teachers, and nearly 30% reported they had been 
given increased responsibility or had been promoted. 

A MODEL FOR EXAMINING IMPACTS OF THE R&D UTILIZATION PROGRAM ' 

As we saw m the preceding overview of selected program impacts, 
the available outcome data are extensive — too extensive, in fact, to permit 
us to simultaneously analyze all of our variables. To reduce the number of 
outcomes to a nlbre manageable set for analysis, we developed a number of 
summary scales through a variety of techniques briefly discussed earlier 
(Chapter 3) and discussed m gr^.ster detail m Appendix C to this report. 
Note, therefore, that the discussion m this section represents very impor- 
tant shifts m the unit of analysis and m the for n of the data being 
considered. The discussion m the previous section was" heavily descriptive 
and largely based on raw data from individual teachers and principals as the 
units of analysis, even when those responfisnts were serving as "observers" 



' *The correlation between principal reports and our field teams' 

reports of organizational impacts was .44 (p<.01), and between our field 
team and teachers .33. 
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of events and conditions at their schools. In contrast, in the present 
discussion of RDU impacts the school as an organization is the unit of 
analysis . To accomplish this shift , summary scales were developed ftom 
batteries of items m the surveys of teachers and principals and in the 
consolidated coding form (CCD. Scales developed t|^om individual teachers 
survey data were then aggr-egated to generate school-level outcomes measures , 
(described in Appendix C). In this section, we identify these outcome 
measures and present data showing how they are related to each other. 



The outcome measures that were developed include the following: 



• Site satisfaction with the ^roblem-solving process , 
based on reported satisfaction with the services 

or activities of the local action team, the RDU 
field agent, the developr^s of adopted materials, 
and the amount of time required to complete the 
process. 

• Site satisfaction with the activities of the 
field agent , including the agent's assistance 
with various aspects of problem solving such as 
diagnosing the problem, developing criteria for 

• selecting a solution, screening potential solu- 
tions, locating additional technical resources, 
etc. 

• Incorporation of the problem-solving process , 
such as reuse of all or part of the activities 
Old procedures which the process involves. 



• Extent to which principal and teachers report the 
problem has been solved through use of the ad^ted 
materials , including improvements in pupil perfor- 
mance, attitudes, and behavior. 



• Impacts on school staff , a global measure 
of personal impacts including increased 
knowledge about curriculum development , 
increased self-confidence, improved teaching 
skills, etc. 



Process Outcomes 




Product Outcomes 



Incorporation of the adopted product and/or 
materials , a measure of the extent to which 
use continues after implementation. 



Unintended Outcomes or Spin-offs 
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• Impacts on the school as an organization , a 

global measure of impact on the school including 
improvements (as a result of participation in 
the RDU program) in curriculum, materials, school 
organization, staff morale, etc. 

We expected that these measures would be interrelated in ways which 
^ would suggest a model for examining program impacts at the site level. 
For example, we predicted that more distal outcomes such as incorporation of 
the adopted materials ^d trte problem-solving process would be a result of 
more proximal or intermediate outcomes such as satisfaction with the process, 
satisfaction with the field agent, and so forth.* 

To investigate this matter, we perform^ a series of stepwise regres- 
sions, using each distal outcome as a dependent measure with the others as 
predictors. These regressions are summarized in "^ible 4-1, which presents, 
standardized regrepsion coefficients for those variables entering as predic- 
. tofs and increasing the R (proportion of explained variance) by at least 
IS, along with an indication of their order of entry. »A raw correlation 
matrix is presented in Table 4*2. 

To graphically summarize how these outcome measures seem to be 
tied together , .we present Figure 4-1, which is a schema of their interrela- 
tionships suggested by the regression results. In this figure, note that the 
outcome measures bo the left of the diagram are those assumed to be mere 
immediate or proximal, while those to the right are assumed to be more diSta]^ 
outcomes. We will first briefly disciTss the model, and then return to an 
examination of the implications for the distal outcomes. \ * ' ^ '\ 

The most immediate outcomes in this '^del are those which are assumed 
to occur closely on the heels of selection and implementation of the adoptc?d 
K4D product. These include two process outcomes — satisfaction with th<; 
activities of the field agent, and satisfaction with the problem-solving 
process and one product outcome — scope of implement atio/i of the R&D product. 
(Scope of implementation refers to the proportion of pupils and teachers in 
the school who are actually exposed to the adopted product and the proportion 
of their school day that is affected by its use.) 

An intermediate outcome, and one which our analyses suggest is 
strongly related to the distal results, is another product outcome — the ex- 
tent to which site staff report the problem has been solved. Not surprising- 
ly, this outcome is strongly related to the scope of product implementation, 
and is a strong predictor of a third produc^t outcome — the extent to which the 
adopted product is incorporated. Product incorporation, a primary aim of the 
RDU program, is also related to the scope of product implei ntation as well 
as to reported satisfaction with the problem-solving process. 



*Other distal outcomes include the extent to which the problem 
has been solved, personal impacts, and organizational impacts. 
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Table 4-1 



STANDARDIZED STEPWISE REGRESSION COEFFICIENTS 
FOR DISTAL OUTCOMES ON OTHER OUTCOMES 
(N = 179 schools) 



Other 
Outcomes 








Distal Outcomes 




• 


Problem 
Solved 


Incorporation 
of R4D Product 


Personal 
Impacts 


Organization 
Impacts 


Process 
Incorporation 


Satisfaction with 
Problem-Solving Process 






.14* 

(3)a 




.16** 

.(4) 


.01 

(3) 


Satisfaction with 
Field Agent 








.15* 

(3) 






Scope of Implementation 




.19* 

(4) 


.32** 

(2) 




.22** • 

(2) 




Problem Solved 






.26** 

(1) 


.29** 

(2) 


.27** 

(1) 




Incorporation of 
Products 




.25** 

(2) 








.12 

(3) 


Personal Impacts 




.22** 

(3) 






.19** 
(3) 




Urnnni 70^100 Tmnoot^o 




(1) 




.33** 
(1) 




.25** 

(1) 


Process Incorporation 










.16* 

(5) 




Multiple T?^: 




.48 


.35 


.32 


.51 


.16 


2 

Adjusted R 




.47 


.33 


.30 


.50 


.13 


a 

Number in parentheses 
* P g .05 


indicates order of entry in 


stepwise regressions. 
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Table 4-2 

PtARSONIAN CORRELATIONS AMONG OUTCOME MEASURES 
(N z \S0 schools) 



Outcowo Measures 



Satisfaction itith 
Problsis-Solving Process 

Satisfaction with 
Field Agent 

Scope of I«ple«entstion 



Satisfsction with SSFisTscFIon Scope of Probiec Incorporat ion Psrsonsl Orgsnization Process 

Problen Solving with Field Implementation Solved of R&O Impacts Impscts Incorporstion 
Agent ^ Product 



.30»» 



.16» 



.35* 



.18« 



.34« 



.12 



.50« 



.30»» 



.27* 



.29« 



.45* 



.26« 



• 27«» 



.11 



.48»» 



.27« 



0^ 



Problem Solved 

Incorporstion of 
RAO Products 

Personal Impacts 
Orgsnizstion Impacts 



.50* 



.4fl«« 



.24* 



.60* 



.46«« 



.49« 



.26« 



.28* 



.21»» 
.39«» 



"^"^p I .05 
p < .01 
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FIGURE 4-1 

ScTMrna of OutcofiM MMtur* Interrelationships 



Satisfaction with 
Field Agent 



Personal Impact, 



Scope of 
Implementation 



Extent to wfifch 
Problem Solved 



Impacts on 
School as 
^ Organization 




Satisfaction with 
the Problem-Solving 
Process 



Problem solution was also strongly related to two outcomes we have 
identified as "spin-offs," since they were not really the intended conse- 
quences of the RDU program. The first of these is a staff, outcome measure — 
reported personal impacts on participants in the problem-solving process. 
This global measure includes reported improvements in teaching skills, 
knowledge, leadership skills, morale, etc., resulting from having gone 
through the RDU process. The extent of personal impacts was also strongly 
related to reported satisfaction with the activities of the field agent, 
with whom the staff worked during the program. 

The second spin-off effect of the RDU program was also strongly re- 
lated to the extent to which the problem was solved. This organizational 
outcome was the global measure of impact on the participating school, and 
includes improved curriculum and materials, decision-making structure, staff 
morale, the school's image in the community, etc. Since the organizational 
impact measure includes staff morale, it is not su ,)rising that it is also 
related to personal impacts on participating staff. 

A second primary aim of the RDU program, along with incorporation 
of the adopted R&D product, is incorporation of the improved problem-solving 
process into school and district decision-making activities. Specifically, 
the RDU program intended that the rational, participatory decision-making 
model it espoused would be utilized repeatedly by the sites to address other 
problems in the future. This outcome, which we consider the most distal 
program impact based on our visits to over 40 participant sites, is most 
strongly related to the globSl measure of organizational impact, and to 
incorporation of the adopted R&D product (the other primary aim of-the 
prbgrant). Predictably, incorporation of the process is also related to 
satisfaction with the process. 

However, incorporation of the process proved to be difficult at 
the site level, and for this reason we suggest it is the Irst outcome of 
the RDU program to be achieved. Our site v^isits strongly indicate that 
the problem with process incorporation lies in the nature of the process 
itself: it IS complicated, time consuming to the point of frustration for 
many sites, and only poorly understood even by its participants. Even in 
sites where there were clear indications that the process (or a part of 
it) was being used again, members of the local decision-making team ex- 
pressed confusion over what they were really doing: "Why are we repeatedly 
prioritizing needs'^" "Why are we spending so much time on this survey of the 
community'^" "I just can't look at another reading program; they're all alike, 
anyway * " F inal ly, even in many sites where staff reported they understood 
what they were doing, they were candid in admitting they could not do it 
again without the help of the field agent (or some other external human 
resource). Since incorporation of the process was a critical thrust of the 
RDU program, its elusiveness is a major problem to which we return ^n a 
later chapter when we investigate the efficacy of various aspects of th^ 
RDU "treatment" in producing impacts on sites. 

The implications of our mode 1 for incorporation of the adopt ed 
program, on the other hand, are clearer. Here in^^orporation is more likely 
if the product 3olves the problem, is 'idely implemented, and is selected 
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via a process that does not. alienate participating staff. More specifically, 
a decision-making process that ensures a close match between the characteris- 
tics of the product and the problem it addresses, followed by widespread 
implementation of that product, increases the likelihood of later product 
incorporation. 

In the case of personal impacts on participating staff, it is not 
surprising that implementation of a product which seems to alleviate the 
problem enhances teachers' feelings of classroom efficacy. Similarly, 
interaction with a competent field agent during a complex problem-solving 
process is likely to increase awareness of R&D resources, bring out leader- 
ship skills, Increase interaction with colleagues, and enhance morale. 

If) the following section, we briefly consider our -model's impliru- 
tions for organizational impacts, since these appear to be enhanced by the 
effectiveness of the product in alleviating the problem, the scope of the 
product's implementation, and the magnitude of the personal impacts on 
part icipat ing staf f . Our analyses suggest that organizat ional impacts are 
also strongly affected by the characteristics of the adopted product and by 
other aspects of the RDU intervention, as we describe in subsequent chapters. 

THE RANGE OF SITE LEVEL IMPACTS OF THE RDU PROGRAM, 

' In deciding how best to measure program impacts at the site level, two 
basic options were identified by project staff. One is a straightforward em- 
pirical approach which involves the development of a variety of scales from 
batteries of items in the surveys of principals and teachers and from the cod- 
ing of case study and site visit data. A second, more typological approach 
was suggested by our increasing familiarity with the sites* experiences, 
gained through site visits and the coding of site visit and case study data. 
This second approach makes use of more global assessments of the kinds of out- 
comes we saw, and is appealing because it reduces the number of different di- 
mensions of program success to be considered in some of our di^*^ussions . 

To develop a typological outcome measure, we focused on four measures 
of program impact: incorporation of R&D products; incorporation of the pro- 
blem-solving process; impacts on the school as an organization; and personal 
impacts on the staff at participating schools. (The computation of these and 
other measures of the RDU program's effects are discussed in Appendix B). 
Note that these include measures of the two primary intended imf^acts of the 
RDU program — incorporation of R&D products and incorporation of the problem- 
solving process — the two areas of spin-off effects we observed — organi- 
zational and personal impacts. The following typology is intended to* capture 
the range of global outcomes found at the sites and provides a concise sum- 
mary of the RDU program's success. Sites were assigned to categories on the 
basis of whether they were "high" (more than one standard deviation higher 
than the mean score), "moderate-to-low" (within one standard deviation of the 
mean) or "low" (more than one standard deviation below the mean) on the four 
impact measures cited above. The resulting categories are defined below 
and illustrated with examples.* 



*As noted in Chapter 3, the validity of this classification was as- 
sessed by whether sites that we were personally familiar with had been allo- 
cated to categories that seemed appropriate based on our site visit materials. 
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• Laryw * * rale. RDU success characterizes sites 
which were given high ratings on all four out- 
comes (product incorporation, process incor- 
poration , personal impacts , and organizational 
impacts) or on any three of the four outcomes, 
thus including at least one program goal and 

at least one apin-off effect (34S of the sample). 

An example of large-scale RDU success occurred in a .traditionally 
all-black southern elementary school. Even before joining the RDU project, 
the school's relatively new principal had sought ways to address the severe 
educational disadvantagement of the school's predominantly low-income pupils. 
In 1976, 88S of the entering kindergarten pupils scored at the 20th percen- 
tile or below on a district -wide inventory of readiness for learning. After 
five or six years of schooling, the children were still far below grade level 
in the basic skills. 

The RDU problem-solving process resulted in the adoption of the 
Wisconsin R&D Center's Individually Guided Education (IGE) program* Fully 
implemented, IGE requires comprehensive change on the part of the adopting 
school. It affects all aspects of school operations, including school 
organization and decision making, instructional programming, curriculum 
and materials, teaching approached, pupil assessmeiit, home-school-community 
relations, relations with school administrative agencies and teacher educa- 
tion institutions, and school-bas^d research and development. The school's 
principal favored IGE over more narrowly focused programs because, as she 
said, "IGE is a process, a total framework. The band-aid treatment will be 
okay after the total treatment has begun." 

When we revisited two years later, the school had changed in signifi- 
cant ways, although its problems were not yet 'solved. The school had been 
completely reorganized into teaching units spanning several grade levels. 
The schedule had been arranged so that the teachers in each unit had time 
together to plan the instructional program on a week-to-week basis to meet 
the current needs of the children in the unit. A great emphasis was placed 
on setting individual goals for the children and helping them to meet those 
goals through frequent regrouping to attend to specific skills. An instruc- 
tional improvement committee made up of the unit leaders played a large role 
in school management and also in continued problem-solving activities. 
Relations had also been maintained with the teacher education center and 
with university professors who had been linked with the school through the 
RDU program. Morale for many teachers was greatly improved. They felt more 
effective in dealing with pupils, believed they had a stronger voice in 
decision making, and felt more unity, cooperation, and concern for one 
another. Although there was as yet no proof that IGE had improved pupil 
performance in tne basic skills, the teachers felt that, in the long run, 
such improvements were bound to be apparent. 

• M ixed-high success sites are those which had 
two high ratings, one a program goal (either 
product or process incorporation) and the 
other a spin-off effect (17% of the sample). 
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M ixed-high success is illustrated by an elementary school m the 
Northwest which participated m the National Education Association's In- 
service Educatitxi Project. The original irnplementation plan was simple— a 
film series called "The Heart of Teaching" would be used as the basis for a 
two-day weekend mservice workshop on the identified problem of teacher 
stress. To everyone's surprise, however, as they began the workshop with a 
discussion of current concerns, they discovered that teacher stress was no 
longer the most important issue facing teachers. Apparently, that problem 
had been resolved through the extensive discussions which took place during 
the process of identifying the problem and selecting and previewing the 
film serieaik What most concerned them now was a lack of consistency in 
expectations concerning student behavior. Instead of going ahead and viewing 
the film series, they spent the remainder of the workshop in a highly produc- 
tive discussion of the student behavior problem and the ways in which staff 
behavior could be used to influence students. 

This unanticipated turn of events set the precedent for intensive 
weekend retreats devoted to brainstorming and group problem solving. These 
retreats were attended by the whole faculty, as well as by the principal, and 
gave them a more formal vehicle for influencing school governance and plan- 
ning. The teachers were pleased to be in control of their own inservice. 
Moreover, since the retreats were focused on problem solving in the school, 
the teachers felt that they had an influence ,over whatever concerns emerged 
among the staff. Although the acceptance of R&D products hao possibly 
declined, the ability to identify and solve problems as a staff had improved 
substantially. In addition, there had been major changes ir teacher morale. 
"The most important thing," said the principal, "is the sharing and trust 
that emerge from the retreats." 

• RDU success charsicterizes those schools 
which had one or two high scores on program 
goals but none on spin-off effect 33 sites, 
or 16% of the sample). 

An example of RDU success occurred m an elementary school in the 
Northwest, which adopted the Wisconsin Design for Reading, the 3unior Great 
Books Program, and an enrichment program for kindergarten pupils. Although 
the Wisconsin Design had been suggested at the district level as the solution 
to a district-wide reading management problem, these decisions were confirmed 
by the school's own task force. 

Two years later, it was reported that nearly 80?o of the teachers were 
using the Wisconsin Design skills tests to determine and record student 
mastery of the various skills. Not as many were using the tests for diagnos- 
tic purposes, and only about hO% were using the supplementary materials to 
any great extent. However, the system had become so thoroughly incorporated 
^ into the routine teachipj process that several individuals had a hard time 
understanding why anyone would be asking questions about it anymore. While 
teachers felt that the systei^ had not made a tremendous impact on the way 
they teach, they felt it had achieved an important purpose, which is contin- 
uity and coordination across grade levels and schools, better record keeping, 
and better identification of student needs. Staff were also quite enthusias- 
tic about the Junior Great Books and kindergarten enrichment programs. 
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In terms of the problem-solving process, there was very little 
permanent change, at least at the school level. The school had a new princi^ 
pal who w?-^ not inclined to participatory problem solving. Hov/ever, at the 
district level, there appeared to be a strong commitment to increasing staff 
inv^olvement in decision snaking. At the time of our last visit, .there were 
three district-wide problem-solving committees, each ^reviewing a different 
problem area. In genei*al, howevef^, teachers did not report high levels of 
personal benefits or changes in the organization. 

• Spin-ofTs are those sites which had high levels 
of organizational an3^r personal impacts but 
which did not adhere closely to the problem- 

' solving or product adoption goals of the program. 

In many cases schools in this category had their « 
own agenda to begin with — e.g., developing curri- 
culum ^uidel ines — and used the resources of the 
ROU program to achieve them ( 10?o of the sample). 

An example of a site which achieved spin-off success is a district 
in the rural south which identified severe scope and sequence problems in 
reading, writing, and mathemat ics through a very^complex needs assessment, 
following that, however, they selected for implementation the Crisp Coynty 
(Georgia) Career Education Package (an adaptation of "It Works"). This 
product obvioU^y was not going to meet all of their neec*s^ and its adoption 
was in fact insignificant compared to the main thrust of the site's probl'cm- 
solving activities — -'thac is, the local development of comprehensive curricu- 
lum guided >n Janguage arts and mathematics. The career education materials 
were simply infused into ' the language-arts curriculum and, indeed, were 
viewed as more or leas optionarl activities. However, the development 
of the curriculum guides was viewed as a milestone isi the school district, 
which had nelfer before g^ven serious attention ^'o tWe coordination of its 
curriculum. 

• Mqderate~t>o-low success characterizes those 
schools which had moderate to low ratings on 
three or four outcorye areas , and no high 
ratings at all ( 10?o of the. sample). 

An elementary school in the South provides an example of mod/ ^ rate- 
to-low succes s. Lack of support from school administrators, scheduling 
constraints which inhib ited broad faculty involvement , attrition among team 
members, and discontinuity in assistance from external change agents all 
helped to hamstring the RDU project in thife school. The team's first defini- 
tion of a problem was rejected by the district administration as a problem 
which could be handled locally without outside assistance. Following a re- 
definition of the prftblem, the team selected an innovation which was then re- 
jected by the project staff as being an unproven product. The team was fin- 
ally resigned to selecting two products from the project's '^knowledge base" 
which they had previously decided did not meet their needs. Following a 
one-^and-a-half day training session on the two products, the teachers made 
a sincere effort to apply what they had learned. However, the brief training 
they had received left tti m somewhat uncertain in their use of the new 
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proqrams. As the vpar ^ore on, the teachers tended to ignore the supplemental 
proqramr as they concerned themselves with getting through their reg'ilar text- 
books. Turnover at the end of the year was high, as^it was in almost every 
vear. By ,)0w, there was just one surviving member of the original facilitator 
team. The priricipal was a passive administrator who gave no particular L»up- 
port to eit' er the problem-solving process or the adopted programs. Given the 
loW adwin^st rat 1 ve support, the lack of resources for additional telcher tram 
ing, the probability of continuing high turnover, and the departure of a ma- 
jority of team members, it seemed obvious that the long-term impact of the RDU 
project would be nothing iiore than having added sorr^ additional materials to 
the classroom shelves. 



• ^ a 1 lure , at lea^^t in project terms, character izes 
those schools which were very poor achievers on 
two or more outcome dimensions, and which h9d no 
^'lqh ratings l^'o of the sample,. 

Ari example of failure is provided by an urban junior high school in 



the f^orth central part of the country. The teachers Wf ? skeptical of 
externally deveiuped products, impatient with the problen;-L.>lv ing process, 
convinced ^hat their own programs were better than most, and preoccupied 
with discipluie prot)lem5:> ar^d witti a pending court decision on desegregation. 
When the> fl.iaiiv selected a product for implementation in the school's 
reading center, it was largely because it closely resemoled what they already 
had. In any case, the decision was moot, since barely a week later it was 
announced that the school woulJ be closed the following ysar, would be 
reopenecJ for e^-'hr^^ graders only, or would be- converted to a K-8 fundamf»ntal 
scriool. For tne ^>"\c <3evf^ral mon^ '•^f , ac^ the school's fate war. being decided, 
it was inpnssible Lo conc-'trat .^n planning a program which probably would 
never be implemented. And, in fact, at ihe end of the school year it was 
announced that the school would be converted to a K-8 fundamental program, 
whil? a n,fijr)rit> of tt>J^ school's teachers and students would he transferred 
to othp" s(^t)ouls. This effect iv el v put an eriri to the project, which m any 
case a^^ eareo to hr qf)ing nownere. 

'he validity of nur typological outcome categories is supported by 
consistency wlt^i other outcome measures and with what was known from the 
study of t^^ scypn oper3tj(jnal prn]ects. for example, examining how other 
{.jtcome merisures were distributed amoni] these categories, we found tha'. sit^es 
rlassir,-^d as large-soale RDU successes also shewed the highest averages on 
measure's Mie scope of R&D product imp 1 ement a i i on , reported that the 
p' -lem rt)ey were addressing through their RDU p'ii t ic iDat ion was solved to 
the greatest extent, and reported the higt^est ^evels jf impacts on pupils. 
These sites also showed the highest mean level of sat ^.sfact iqn with the 
problem-solving pr{j{:ess, and were the most satisfied with the activities of 
the field agent. in addiMon, they had the highest percentages of staf^^ 
reporMnq that PDU was quite different from previous prohjem-sn] v iru] prar- 
t r^es. 
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SU^fWRY AND CONCLUSIONS 



This chapter addresses the question of vfhat happened in the RDU 
program. We have seen that RDU appears to have had a variety of positive 
impacts y not only m its intended areas related to the use and incorporation 
of new curricular products and materials and an improved problem-8olvin;j 
process 9 but also in two areas we identified as spin-offs. These spin-off 
impacts included positive effects on participating staff and on their schools 
as well. We have also seen that the various outcomes we identified may be 
interrelated in ways which provide a mod^l for examining program impacts. 

What we have not yet discussed is wtiy these findings occurred. In 
particular, we have not presented any evidence that the various elements of 
the RDU "treatment" — the products, the problem-solving process, and the use 
of external human resources — s e oirectly related to the magnitude of these 
effects. Our discussion of the "wh*ys" is presented m the remaining chapters. 
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CHAPTER 5 



THE ROLE AND IMPACT OF EDUCATIJQNAL "PRODUCTS" 
IN THE RDU PROGRAM 



INTRODUCTION 

The intent of the RDU program, as it was expressed in the RFP 
(NIE, i973) and other early documents, was to help schools solve educational 
problems through the use of existing research and development-based "pro- 
ducts" (curriculum innovations). The original RFP indicated that proposed 
projects should limit their knowledge bases to products related to either 
basic skills or career education . In addition, the RFP indicated that 
contractors should emphasize quality control over the prnHijcts, Th$ "i/oov^- 
ledge base" or product pool for each of the seven projects was to be^<ttfve- 
loped separately, and was to emphasize proven ef fectivenes.s and evidence of 
transportability from one site to another. Thus, throughout the early 
portions of the RDU program a great deal of attention within each project was 
given to developing a -knowledge base that contained acceptable products and 
designing a process for delivering these products to the schools. In addi- 
tiorf, the RFP implied that the projects should attend to some process of 
matching each site^s needs with available products. While the program 
designers were cognizant of ongoing , research which argued that adoption was 
insufficient to guarantee school impact, they reflected the conviction that, 
if the innovation adopted did not meet certairt criteria, implementation would 
be pointless. In the remainder of this chapter, we address questions re- 
garding both the adoption process, and the impact of product characteristics 
upon J<nowledge utilization (implementation) and other school improvement 
outcomes. 

MATCHING SITE NEEDS WITH AVAILABLE PRODUCTS^ 

The first stage of any site's participation in the RDU program 
involv::d 3 needs assessment of some type. Assuming that a site had completed 
a needs assessment, it was then ready to consider various products that could 
be adopted. This matching process war, a potentially difficult one for the 
RDU projects and the schools they served, who were not experienced in evalua- 
tihg the quality and applicability of t?xternally developed materials. Most 
of the seven projects ultimately de\ eloped a general procedure whereby lists 
of potentially relevant products from the knowledge base were frade available 
to the eite teams for their consideration. However, there were significant 
variations among the projects in following this procedure. 



^For a more extensive discussion of thjs topic, see Yin, Gwaltney, 
ana louis (1980). 
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Modes of Communication 

In four cases (Florida, Georgia, Michigan, and NEA), the site teams 
were encouraged to make direct contact with knowledge-base starr,* indicating 
the topic or topics of interest that resulted from their needs assessments. 
The knowledge-base staff then identified an appropriate array of potentially 
relevant products and sent brief descriptions of these products to the 
sites. In the NEA project, this was done when a *^ite would make a telephone 
call to the knowledge-base staff and, descriptions of all the products m the 
relevant categories, based on a prior classification scheme, were then sent 
to the site. In other cases, and during the initial phases of most projects, 
the procedure was more interactive and involved face-to-face communications. 
With the site staff visiting the knowledge-base staff or vice versa. In 
Georgia, some initial orientation to the whole product array m the knowledge 
base was communicated through two "educational exchanges," or conferences at 
which teachers from many sites were invited to review a wide array of mater- 
ials. These conferences were uniformly judged to^be a highly satisfactory 
way of orienting site personnel and giving them an idea of the potential 
products be fore the needs assessments were completed. 

In the three other cases (The NETWORK, NRC, and Pennsylvania), the 
site teams did not make direct contact with the knowledge-base staff, but 
worked instead with the p^rojects' field agents. In these cases, the agent 
ascertained the topics of interest from the »ite, worked with the knowledge- 
base staff to select potential innovations, and then explained the various 
possibilities to the site personnel. The field agent thus served as an 
intert»iediary m the matching process. 

This major variation in communication links did not appear to create 
any consistent differences in the ultimate product adpp*tion patterns, but 
deserves further attention because of the different roles implied for the 
field a<5ent . In the first mode of comrminicat ion, where sites dealt di- 
rectly with the knowledge-base staff, the site personnel were regarded as 
the primary users of the knowledge base, and knowledge-base documents were 
oriented toward the terminology and need*= of practitioners. The field agent 
played only a secondary role in the communicat ions process, generally 
being informed of. the site's interaction with the knowledge-base staff after 
it had occurred. In at least one project (NEA), field agents came to play 
increasingly peripheral functions as » result of this procedure, and on 
occasion the field agents were not even informed about the site's comm»jni- 
cation with the knowledge-base staff. In the second mode of communication, 
where sites worked through the field agents and only indirectly with the 
knowledge-base staff, the field agents were regarded as the primary users 



♦In Michigan, sites contacted the knowledge-base staff m the 
state department of education rather than in the sub-contracted agency. 

**A "mixed" mode of communications could also occur on occasion 
^e.q., in a few sites m Georgia and NRC), inwhich the knowledge-base 
staff and field a? ?nts worked together m dealing with site personnel. 
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of the 'knowledge base. In this situation, knowledqe-base documents were 
oriented toward the terminology and needs of the field agents, and thus the 
field agents had a strong role in influencing a site's final selection. 
Overall, .the mode of communication reflected the degree of activity on 
the part of the field agents; where they were not heavily involved, the 
knowiedge-base staff actually filled the Unking function. This was the 
ca^e for several of the projects. 

Size of the Candidate Array 

Each project also had to develop its own sense of the appropriate 
size of the initial candidate array, whether presented directly to the site 
or through a field agent. The size, or number of products, had to be large 
enough so that sites could have some choice m making their final selection, 
but small enouoh to be manageable. Projects generally presented about ten 
products, where available, in this initial array. This pr'-blem was further 
compounded by the nature of the materials used to describe each product. 
Most Sites would have preferred receiving the product itself, but would then 
have had difficulty in reviewing the materials for such a large array. As a 
result, all the projects developed their own one- or two-page descriptive 
summarte^s of each project, and this was the material that was sent to each 
site.*^ 

Screening of Candidate Products 

Site^' screening of candidate products generally followed two stages. 
During the first stage, the site would, m theory, review the candidate 
products for their potential relevance, and reduce the initial pool of 
candidates to a smaller set. of two or three final candidates. During the 
second stan^, the site would reguest mo-e information about these final 
candidates, and the know! edqe-base staff would have to be contacted again 
for this inform.it ion. At this point, the knowledge-base staff typically 
loaned the actual product materials to the site, or even suggested direct 
contact between the site and the onainal developers of thp products fe.q,, 
PennsvKani« J . The site teams t^en reviewed }n detail these final candi- 
dates and selected one f^.r adoption. ♦ 

This second stage was conducted most system^t icaj 1> in t wa of the RDU 
projects 'Georgia and l«f?nnsy Ivahia) . Occasionally, however, the knowledf]e- 
base staff had dif^firulty keeping track of the product materials that h.^d 
been loaned out. In the other projects, the second stage was often blurred 
With the first, so that the process of narrowina down the initial list of 
candidates occurred in a less distinctive, two-stage manner. For instance. 



*The issue of the size and nature of materials m the initial 
arrav needs to h" qiven greater attention lO: the future. Pennsylvania, 
for example, I'.tially presented a large number of candidate products 
to sites. Wh- I «iites were finding it difficult tj select a product from^ 
surh a large ruray, the knowledge-base staff significantly reduced the 
number of candidate products that they presented. 
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Mhere field agents were heavily involved m the matching process, the two 
stages might have been collapsed into a single stage, oi several iterations 
might even have occurred befqre a final adoption was made. t 

Whatever the process, the final selection often revealed a host of 
problems that had to be addressed m a site-by-sxte manner* First, in 
some cases the sites may have had a single product m mind all along, pos- 
sibly preceding the needs assessment activity. This prior preference may 
have been well-suited to the site's own aense of its needs before entering 
the RDU program, or it may have reflected a bias that was not justified by 
the site's actual needs. 

Second, m othe." cases the sites may have identified their own 
candidate products to augment the candidates selected by the knowledge-base 
staff. This was especially true m those projects (e.g., Michigan, Georgia, 
and NEA) where it took two or three years to complete the knowledge base, 
and where the sites had therefore advanced more quickly than the fiOU pro- 
jects' preparations. In these cases, the sites* candidates were frequently 
incorporated into the incomplete knowledge base. These cases represented 
important occasions when nonvaiidated products could be incorporated into 
the system because the sites did not necessarily present any compelling 
evidence that their candidates had been validated for prior effectiveness. 

Third , the Vev lew of candidate products often revealed a mismatch 
between the categories or terminology used by the knowledge bases to classify 
their products and the categories or terminology used by the sites. The 
level of specif).city could be different. Thus, sites could decide in their 
needs assessments that they had a "motivational" problem among the students, 
which was not specific enough to identify accurately the potentially rele- 
vant products (e.g., Florida); further probes were needed to determine that 
the specific problem may have beep students fiqhtmg in the hallways, and 
on this basis it was easier to determine whether a relevant product was 
available or not. Conversely, some sites came up with specific needs state- 
ments, but then could not easily cope with the generality of the product 
descriptions, (e.g., Pennsylvania). Similarly, some sites made their needs 
known in terms of curriculum content even though the knowledge-base products 
were initially classified by teaching processes (e.g., NLA), or vice versa 
(e.g. , Floridp) . 

Fourth, it was entirely possible that*none of the candidate products, 
even with accurate communications, served a site's needs. In theory, the 
knowledge-base staff was then supposed to conduct a further search, bpyond 
those products that were included m the projocts' formal knowledge base, for 
a potentially relevant product — a provisi'^n that was covered by the original 
RFP (NIE, 1975:15). However, insufficient attention had been given to 
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the fact that this wider search could take a long time — far exceeding the 
site's schedule for adopting a specific product — and this broader search 
would have to be conducted with undue haste, again leading to the possibility 
of using nonvalidated products. Whatever the outcome, the provision for 
having a further iteration at this point was judged unrealistic by some 
projects • 

All of these problems should suggest that the screening process was 
not an orderly or simple one. In fact, we believe that this process bore 
the brunt of one of the conflicting elements in the basic design of the 
RDU program.* On the one hand, sites were to use existing R4D products. 
On the other hand, sites were to undergo a problem-solving process, where- 
by a needs assessment was the initial step.** Only unabashed optimism 
would lead to the conclusion that the available validated products were 
likely to match, with high frequency, the articulated needs of sites. For 
example, ir one* project that carefully documented the matching process, it 
was noted that the project was unable to fii^d acceptable prouucts in 405o of 
the "phase I" sites. As a partial remedy, thg» RFP did make one provision for 
dealing with the potential conflict: \ 

...a legitimate project outcome could be the conclu- 
sion that in a specific local situatioh there is no 
RAD (product] that represents an acceptable solution 
to the defined problem. Such a conclusion, properly 
documented, could add to [NIE*s] understanding of 
field requirements for further R&D. (NIE, 1975:4-3) 

This provision fails, however, to indicate what the site should have done 
when this situation was encountered. Most, if not all, of the sites were 
recruited into the RDU program on the basis that some assistance would be 
provided in dealing with their school problems. An impasse uf the sort 
described in the RFP, while potentially useful to NIE, would not likely have 
been an acceptable conclusion from the site's point of view. Not surpris- 
ingly, the seven RDU projects theref',re did everything they could to firxi 
some acceotable product for every site, even when the impasse was encoun- 
tered. In the above named project, for example, the response ir\ most cases 
was to allow the site to adopt a product that they located through other 
sources, as long as it was reviewed and found acceptable by the project. At 
the same time, it is also true that most projects did little to document^ on 
a systematic baeis, those situations in which the impasse occurred, and thus 
there is only sparse information regarJinq further needs for new types of 
products. The only information of this sort derives from interviews with 
project directors or knowledge-base staff », who typically reported the need 
for more products in secondnry education in general, and in non-readinq and 
non-math curriculum topics ift particular. 



*There is evidence from related intervi*»ws that some NIF staff 
members were aware of this potential conflict but did not influence the 
design of the RDU prograr^r 

**fhe conflirt between these elements m?»y be found in programs other 
than education ^e.:,, see Yin, 1978; and Roessner , 1979'. 
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Final Adoption Patterns 

As a result of this matching process, the sites finally did adopt 
some product. For sites that hpd adopted a product by the Spring of 1979,* 
the full list of adopted products, by project, is shown in Appendix 
Sixty-four products were adopted by only a single site, whereas 36 products 
were adopted by more than one site; of these multiple adoptions, the most 
popular products were: Wisconsin Design for Reading {adopted by 11 sites); 
Career Development Centered Curriculum (seven sites;; San Diego R2R (seven 
sites); Exemplary Center for Reading Instruct ion--ECRI (six sites); It 
Works (six sites); AEL Career Decision Making Program (six sites); and 
Houghton-MiTf lin Basal Management System (five sites). 

This process of atterpting to meet sites* needs with a predetermined 
set of products inevitably resulted iil some loss of emphasis on the dimension 
of quality control. While it is impossible to determine precisely whether 
ftach adopted product had been validated at the time it was adopted, several 
approaches to assessing the impact of the matching process on adoptions were 
used* 1 he first involved using project records to estimate the percentage 
of adoptions that were outside of the defined knowledge base of each project. 
Overall, 21% of the adoptions did not involve products that had been identi- 
fied by the site/project as exemplary. Again, using project records, we are 
also able to estimate on a very rou^- basis the number of adoptions that may 
have involved products that did not meet the criteria that the program 
initially advocated. Table 3-1 presents the information from project rtcords 
regarding the validation status of the adopted products, 

A notable feature of the table is the preponderance of NDN products 
over NIE sponsored products. While we have not examined products to de- 
termine the reasons for this, both sites and project directors commenced that 
NDN products were often more easily available, came with better (and often 
free) training, and were more likely to have adopter sites nearby, which made 
them easier to visit during the solution-selection process. The fact that 
they were "practitioner developed" does not appear, according to these 
sources, to have been a factor m their popularity. 

Summary 

The discrepancy between what the projects intenrled and what they 
were able to effectively provjie to sites was s result of two primary fac- 
tors. First, in many areas there were simply not a sufficient number of 
formally validated educational products to meet the needs of local schools. 
Among the topics treated by the RDU projects in which few field-tested or 
externally validated products were found after considerable effort were the 
following: career education; district or sohool-wide planning; and inser- 
*'ice training for teachers. In addition, the area of secondary school 
curricula, particularly m tt^e area of basic skills, produced very few tested 
educationcl products that met school needs. In order to be responsive to the 
service delivery mandates of their programs, the RDU projects were forced in 
many instances to use expert judgement on surface validity, lather than 
external evaluation data as the quality control basrs m admitting products 
to their pools. 

"^*Hany sites had not reached the adoption step by that time. 

80 

er|c '^f 



Table 5-1 

PRODUCT ADOPTION PATTERNS FOR EACh RDU PROJECT, BY SPRING 1979 



ADOPTIONS 



Assumed To Have Been Validated 



Not Known To Have Been Validated 



Project Total 



Nurrt)er 
from 
NIE 
Catalog 
(Validated) 



Nuntoer 

from 

NON 
Catalog 



Nufrtber 
Through 
Local 
Validation* 



NRC 


45 


4 


5 


6 


< Pennsylvania 


13 




9 


3 


Georgia . 


24 


1 


5 




Network 


27 


6 


18 


3 




7 


7 


7 




Florida 


22 


4 


1 




M'chigan 


63 




9 




Total 


194 


15 


47 


12 



NufTber 
from NIE 
Catalog 
(Not 
Validated) 



Number 
Outside 
of 

Knowledge 
Base** 



30 
1 
4 



3 

38*^ 



Nuntoer 
Without 
Local 
Validation** 



Percent 
Non- 
Validated 





66.7 




7.7 


13 


75.0 




0.0 


7 


7 


17 


77.3 


48 


85.7 
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Each of the projects with a local validation procedure was arbitrarily assumed to have properly tjsed 
It, eading to a mure conservative estimate of the overall proportion of nonvalidated product-.. The 

dat"?io p^Sdl?".'" '''^ ""'^ P"^""^' -5° l^'^^l 

"?f;r.xL^f^nf validated through an alternative procedure (e.g.. by the commercial publisher] 

me extent of this phenomenon is not known. 

"'I^nnfJ'"'^"''*'^ Ju^*' r""^ outside the knowledge base were nevertheless NIC or NDN prooucts. These three 
products were, therefore, assigned to the NIE or NDN colutms in this table. 
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Second, in the case of the 205 of the sites that chose products from 
outside the knowledge base (and the many more that adopted an approved 
product in order to satisfy the project, but also found non-approved products 
to supplement this choice), site preferences and the overriding norms of 
service delivery were the key. In most cases, these "low quality" selections 
were made after six months to a year of intensive invdlvement between the 
site and the project. It is small wonder that projects were inclined to 
continue delivering services to schools that were committed to innovation and 
to the project, but simply could not find an approved product to meet their 
needs . 

Having determined that the matching process Is problematic, it 
is also important to note that the RDU project was, on the whole, enormously 
successful m providing the sites with some product that had the potential 
for meeting the school's expressed needs. As we pointed out m Chapter 
4, most schools adopted something, and of those who adopted, most expressed 
satisfaction and were committed to utilization. However, as noted also 
in Chapter 4, there was considerable variance on many of the variables that 
we have used to serve as indicators of ^he longer range impacts of the 
RDU program on the school. The question^o which we now turn is, what is 
the impact of the product's characteristics upon more long-r^nge school-leve 
outcomes'^ 

DOES ME SOLUTION MAKE A DIFFERENCE: A CLOSER 
LOOK AT THE CHARACTERISTICS OF ADOPTED PRODUCTS 

While the projects found that the process of developing knowledge 
baf^es and matching local site needs to available products was difficult in 
many instances, the problems of managing a knowledge base were limited 
compared to those of managing a dispersed staff of field agents and training 
subcontractors, or managing the internal problem-solving process used by 
individual schools < Louis and Rosenblum, 1981). The fact that it is theore- 
tically possible to control the products and materials in Che knowledge bases 
makes an analysis of product characteristics and their impact on school 
improvement outcomes particularly relevant for the development of policy and 
management recommendations. Befoie describing t^e impact of products on 
local site outcomes, it is useful to review Some of the major perspectives on 
the role of products in managed change programs, and their implications for 
the measures of p»"oducl characteristirp* used in this report. 

Three Perspectives on Products and Outcomes 

There are a variety of traditions that reflect somewhat different 
views of the product characteristics that may have the most impact upon 
school improveiient : the diffusion perspective, the adopter perspective 
i the adaptation perspective. In selecting measures to include in the 
following analysis, we have attempted to reflect each of these. 

Diffusi on Perspective . Perhaps the most traditional of these is 
di ffiTsi^orr research model which has typically attempted to classify 
objective characteristics of the innovation, and determine **.eir relationship 
to the spread of that ir.novation in a general population (Rogers and Shoe- 
maker, 1971). 
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Among the product characteristics that are more frequently measured 
m this highly technological perspective are the complexity* of the innova- 
tion (the number of different parts that it has), the tr ialability or reversi- 
bility of the innovation (how difficult would it be to stop using it if it 
proves unsatisfactory), and the relative advantage compared to existing 
practices. In addition, other research has indicated that a key problem in 
the spread of educational innovations is transferability, which often^trans- 
lates into whether the innovation is accompanied by adequate guidance for 
implementation by another person (Stearns, 1973). In addition, current 
federal policies strongly support the notion that a key to obtaining benefi- 
cial sciool outcomes is disseminating only field-tested ^ r validated products 
(although there is considerable evidence that practitioners use criteria of 
source credibility rather than research evidence to evaluate information). 
As noted in the previous section the latter two characteristic? of products' 
were part of the general guidelines that NIE developed to assist the seven 
RDU projects m developing their knowledge bases. 

Adopter Perspective ^ A second way of approaching the importance 
of product characteristics is most distinctively expressed by Downs and 
Mohr (1976) who assert that innovation characteristics can only be under- 
stood by looking at their meaning from the perspective of the potential 
adopter . Thus, this approach would tend to examine not objective complexity, 
but a subjective measure of how difficult it is for the individual to adopt 
it , and how much change must occur for full implementation to take place. In 
addition, it may be argued that the objective characteristics are less 
important than the degree to which there is a match between the expressed 
need or problem of r-n adopting school or individual and the ostensible 
objectives of the iruipvative program. Finally, it may matter a great 
deal whether the adopting school or individual believes the product to 
provide genuinely new and better ways of doing things that are relevant to 
the problems of the school and classroom. This might be labeled subject ive 
quality . 

Adaptation Perspective . A final perspective is derived from the 
emphasis upon adaptation and local development espoused by Mann (1979) 
and Berman and McLaufjhlin (1977). From this perspective, one might argue 
that the objective and initial subjective reactions to an "innovation" are 
less important than whether the local. staff takes an active role in reworking 
the materials and ideas to fit the local context. That JS, the key to a 
"good" innovation is the existence of l ocal materials development and formal 
adaptations of the externally developed product. 

Data Sources and Analysis 

F,afh of these three perspectives was *:apped in measures of product 
characteristics used in the study of the R^cD Utilization program. Most 
of the product ch ar icter i st ic s-~ f le 1 d-t es t/v al idat ion status, relative 
advantage, complexity, reversibility, provision of guidance for implementa- 
tion, the cT-nount of pre- and oost-implementat ion adaptation, and the degree 
to which the product appeared to match the identified problem in the school — 
were measured through single questionnaire items on the consolidated coding 
form (CCF) completed by site visit staff members. The perception of product 




♦Underlined words represent measures used in the analysis, 
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quality was composed of an index of three 'items from the teacher question- 
naire reflect inq the novelty and applicability of the ideas to the individual 
teachers and school context, while the difficulty of implementation was an 
index composed of four highly interrelated items (amount of change required 
from previous teaching style, amount of change required in classroom organi- 
zation or management, amount of recordkeeping required, and overall diffi- 
culty of implementation), * 

To assess the importance of product characteristics in explaining 
school-level outcome measures, we performed a series of stepwise regression 
analyses, simultaneously relating sets of indicators of product characteris- 
tics which reflect all three perspectives on products and outcomes to six Uiy 
school-level outcomes. The outcome measures included the extent to which 
the problem was reported as ^'solved,'* the extent of program impact on the 
school as an organization, personal impacts on participating staff, incor- 
poration of the program, and incorporation of the problem-solving process. 
We also examined the scope of product implementation — i.e., the extent of 
product use. The results of these analyses are summarized in Table 5-2, 
which presents standardized stepwise regression coefficients and propor- 
tions of variance in outcomes explained by the products* characteristics 
(R s). In the discussion which follow^;, we first examine the importance 
of product characteristics from the diffusion, adopter, and adaptation 
perspectives (the rows in Table 3-2). We then look more closely at the 
explanation of specific outcomes (the coltJTnhs in Table 3-2). 

The diffusion perspective is represented in Table 3-2 by five indica- 
tors of product characteristics, as discussed above: complexity, reversi- 
bility, relative advantage over previous practice, availability of adequate 
guidance for implementation, and product validation. Our analyseF**suggest 
that some of these product characteristics were strongly related to several 
outcome measures. Product complexity (in the sense of the number of things 
which must change in order to implement the product) is important in explain- 
ing the school-level outcome measures. Complexity is strongly and positively 
related to organizational impacts on the school, incorporation and continuing 
use of the product, and personal impacts on staff. Not surprisingly, the 
product's relative advantage over previous practices was also positively 
related to product incorporation, but was not significantly related to other 
outcome measures. Product validation was positively related only to process 
incorporation. Neither rever'^ibi lity nor the availability of adequate 
guidance for implementation was significantly related to any of the outcome 
measures in these analyses. 

Thus, the fusion perspective appears to have somr validity <or 
explaining four of the six outcomes: organizational impacts; product incor- 
poration; process incorporation; and personal impact on staff. Note also 
that product validation^^ an indicator of this perspective, was the only 
product character ist ic sign if icant ly related to our most elusive outcome, 
incorporation of the problem-solving process. The diffusion perspective was 
of little utility in explaining the nnore proximal outcomes: the extent to 
which the probierri was solved or' the scope of product implementation. 



Table 5-2 



STANOATOIZEO STEPWISE REGRESSION (BETA)* COEFriCIENTS 
rOR THE RELATIONSHIP BET>CEN PTOOUCT CHARACTERIS-:CS AND STX 
«ASURES Of SCHOa OUTCOMES 

(N s 60) 



Product 
Variables 



Hiltiple 

Adjusted Multiple 



^rqsnizational 
Impacts 



Diffusion Psr s pective 

Product Cosipls»ity ,31*« 
Prouuct Ravarsability 
Ralativa Advvttaga 
Ada<|uate Inpls*. Guidance 
Product Validated 

_^doptar Perspective 

Difficulty of Implenientation .28»* 
Match to Problem 
Product Quality 

Captation Perspective 

New Materials Development -.i? 
Pre- Inp lenent • t ion Adapt . 
Post -Imp lenient St ion Adapt. 



.34 
.28 



School Outcomea 



Product 
Incorporation 



Process 
Incorporation 



Problsffi 
Solved 



,29« 

.20* 



.19 
.24** 



.46 
.40 



.20 



.1? 
.10 



.17 



.23* 

.13 

.58* 



-.19 



.46 



Scope of 
Imple«Bentation 



Impacts 



.22 



.31* 



.19 



-..27* 



.33 
.26 



.21* 



.36- 



-.16 
-.13 



^36 
.30 



* vanables .hich contribut,^ to th. reported multiple r\ 

ailed to increase the Muitipl* R by l\ or 



..... . *w*^.,v^ M*«^rFiiBa oruv ror tnose variables which 

ihe selection process was stopped when additional variables fax 
the brder of entry was ur^forced. 



or more? 



•♦pi 



.05 
.01 
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The adopter perspective is represented in these analyses by three 
product characteristics: difficulty of implementation (e.g., changes in 
classroom organization, record-keeping, and other procedures necessitated by 
implementation); the degree to which the adopted product matches or fits the 
problem it was selected to solve; and the teachers* assessment oF product 
quality (i.e., the extent: to which it provides- new information, meets a 
classroom need, and seems relevant to pressing school problems). 

Teachers' assessments of difficulty in implementing the product were 
positively related to three outcomes: organizational impacts, the extent to 
which the problem was solved, and scope of product implementation. Product 
quality was significantly related to product incorporation, the extent to 
which the problem vfas solved, and personal impacts on staff. Surprisingly, 
the product's match to the problem was not significantly related to any of 
the outcomes. 

♦ The adopter perspective, therefore/ also appears to^ have some valid- 
ity in explaining school outcomes.* Note particularly that this perspective 
was more useful , than the diffusion perspective in explaining the proximal 
outcomes, ^cope of implementation and extent to which the problem was solved. 

The adaptat:^on perspective is represented by three product character- 
istics: the neecJ to develop additional new materials in order to implement 
the product; the extent bo which the product was adapted prior to implerpenta- 
tion; and the extent to which it was adapted after implemerrtation. In the 
context of our. findings already discussed and the work of others, our inves- 
tigation of the adaptation perspective was particularly interesting. 

^ The need for adaptation of the product, whether prior to implementa- 

tion or subsequent to it, is part oF the tradition of "mutual adaptation" 
explicated m other research (Berman and MacLaughlin, 1975). Our findings 
run contrary to ^this research^ as Table^5-2 indicates. Four different 
indicators of the' product's need ^f or adaptation were included in our analy- 
ses: two items m the teachers* survey asked about the extent to which the 
products required modification, and the extent to which they required 'local 
development of materials. In addition, two items m the CCF data provided 
information on the extent to which the products were adapted (modified) prior 
to and subsequent to implementation! The teachers' assessments of the need 
for product modifications did not enter any of the regression^ on school 
outcomes (and are not included on TLble 5-2); the teachers' assessments of 
the need for local development of mater ials 4ihd the two CCF items on the need 
for modifications did enter some of the regressions with three school-level 
outcomes (problem solution, impacts on the school as an organization, and 
impacts on staff), though their individual regression coefficients, were not. 
statistically significant at the .05 level (alpha's ranged frcjn .06 to .18). 
However, the results suggest tliat such adaptations m^y be negatively related 
to school outcomes. That is, adapting the proi:lucts appears to reduce their 
efficacy. This may also reflect the nature of the solution selection acti- 
vities, however, raising the que%t^n uf whether products which really 
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matched the site's problem were identified and carefull/^creenpd before Lhe 
final selecfion was made. We hypothesize that more careful selection proce- 
dures could substantially reduce the need for adaptation before and after 
implementation and enhance the other program outcomes (a topiC'that w^ll be 
explored further in Chapter 9). 
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In addition, 'we find tjiat on-site materials development is begat ively 
related to t^e scope of the implementation occurring an schools. In other 
words, high levels of local intervention in the RDU program appear lo accom- 
pany smaller scale, more localized activi*:ies involving fewer teachers and 
less time. This finding is not "^unreasonable — RDU was a teacher-dominated 
process for the most part, and in few schools can teachers commit the time 
needed to design large-scale change programs "from scratch.*' It also sug- 
gests, however, that with a limited amount of resources, they may be turned 
either toward broader implementation or toward on-site materials development, 
but probably not both. 

,\ 

Thi« se*' of findings also raises an interesting question with regard 
to the now common assumption that mutual adaptation is desirable. We suggest 
that in, the more top-down approach to change in educational organizations, 
adaptation of the adopted R&D product might serve the useful function of 
providing the teaching staff, who will be expected to implement the product, 
with a sense of ownership. This feeling of personal investment in the change 
program might otherwise be lacking, depending on just how central ized--e.g. , 
in the offices of district or building administrators — the decisions regard- 
ing the change program , real ly are. Given an increased sense of ownership,i 
teaching 'st aff may then implement and continue to use the product to a 
greater extent than would otherwise have been the case, and may feel more 
positive about its efficacy regardless of how Iheir adaptations might have 
^Itered the product's quality. 

In contrast, the R&D Utilizat/lon pxogram, with its emphasis on a 
bottom-up decision-making process, aimed at maximizing teacher involvement in 
careful selection and ownership of the program at every step along the way. 
Thus, it may not have been necessary to involve teachers in the adaptation of 
the product when, given careful selection, it really needed no such modifica- 
tion. This contention is supported by our findings that local development of 
materials and other adaptations, both before aiSql^after implementation, were 
neqat ively related to program ^outcomes . 

In summary, therefore, we found that both the diffusion and adopter 
perspectives on product characteristics and school outcomes were supported as 
explanations for the RDU impacts in schools. The adaptation perspective, 
however, was contradicted. 

A review of the columns o^f Table 5-2 reveals the combinations of 
product characteristics that appear to be the. most favorable for achieving 
each of the school outcomes. Positive organizational xmpacfS on school^ 
are maximized wf en complex products which are dif f icul t To implement are 
selected. Site visit data suggest that such products may require more inter- 
act ion among st aff and with administ rators m the implement at ion process-- 
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which results in iiDproved teaching methods. In a number of instances, more' 
complex products were sufficiently "newsworthy" that srhools were able to 
gam effective publicity for their new Activities in local papers, improving 
their image in the community. More complex products also provided a more 
substantial basis for "spin off" activities involving additional innovation 
or change that was not originally part of the intended program. 

Product incorporation {yr^'r; continued use) is more likely to be 
achieved when the product selected is complex, has clear -advantages over 
previous practices^ and is of high quality .v Although -a close match between 
product and the problenf it was chosen to address glsd seems to erthance 
chances for intxjrporation, the relationship was not atatiaftically significant 
in this analysis. This cluster of variables tr.ids tb suggest that incorpora- 
tion has a strong element of rationality-: products which are viewed as good, 
and are well suited to their local environment, will tend ,to be retained. It 
IS important to emphasize that this interpretation does not imply that 
rationality dominates the incorporation decision, but only that there appears 
to be some match between the effectiveness and quality of the program and its 
retention. Again, the reasons for this may be the emphasis on rational 
decision making and the participation of all affected parties throughout the 
RDU process. In addition, site visit data indicate that the level of actual 
in-kind investments in RDU schools may have created a built-in incentive for 
'incorporation that apparently dio not exist -in many of the programs studied 
by Berman & McLaughlin (1977). (See Chapter 10 for a discussion of program 
costs and in-kind contributions.) 

Staff development or personal outcoirles also seem greatest when a 
complex product of high subjective .quality is. chosen. As was suggested 
above, tiie more complex products — i.e\, those requiring more changes in 
previous pract ice--may have positive effects on staff because their implemen- 
tation often required more interaction among colleagues. In addition, many 
staff mennbers who were interviewed suggested that their sense of efficacy 
was increased when, as a faculty, they were able to implement a complicated 
new curriculum program. Increasing levels of interaction around, the imple- 
mentation of a complex new set of prartic^es also seems to increase teachers* 
perceptions that they have something t^ offer one another. Complex products 
typically required more planning fcr implementation, thus exposing teachers 
more systematically to situations where they learned more about new curricu- 
lum ideas, and decision-making or problem-solving skills. 

The extent to which the problem was solvegj is the best predictecj 
outcome of all our outcomes measures, with 46?o of the variance accounted for 
by five variables. The key predictors,, of problem resolution were product 
quality and difficulty of implementation, with lesser contributions made by 
the adequacy of implementation guidance, the degree to which the product 
matched, the problem, and the minimization of post-implementation adaptation. 
This cluster of variables suggests considerable support for th ? interpreta- 
tion of Gross et al.'s (1971) study of the problems of implementation at the 
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Cambire elenjentary school. The authors distinguish between two different 
kinds of "difficulty of implementation." The first derive^s from the 'amount 
of change that is requested of the teacher — the more he or she must move frcin 
current ^practices,' the more difficult the implementation. A second, source 
of difficulty emerges from the vaguenese of expectations surrounding the 
proposed ^changes, and can oe summarized by whether the^ teachers can actually 
understand what they are expected to do. While the. fiVst type of difficulty 
of implementation — amount df change required — improves the chances of problem 
resolution, the second type — vagueness and appropriateness of implementation 
requirements — militates strongly against it. 

We now turn to tte explanation of incorporation of the problem- 
solving, process . As the n for this column shows, proems incorporation 
was a relatively difficult progrlim outcome to explain: product characteris- 
tics, though they were generally powerful predictive ^variables, only ex- 
plained 175 of the variance in measure of process incorporation. Further- 
more, the nroduct characteristics, variables which entered the regression 
with process incorporation were a different set from those which had been 
useful in the analyses of other outcon»es. Process incorporation was highest 
at those schools which adopted field-validated products that showed a clear 
advantage over previous practice. On the other hand, product quality and the 
amount" of effort required fo.r implementation seemed to be irrelevant^ though 
these had been important predictors of other outcomes. (We will see in a^ 
later chapter that process incorporation was more strongly related to 
characteristics of the sites than to aspects of the RDU intervention itself.) 

ARE GOOD PRODUCTS ENOUGH?, 

While ppoduct characteristics have a considerable impact on the scope 
of implementation in a school (Tabl& 3-2), we must be careful to avoid a 
deterministic interpretation of this finding. We do not know, for example, 
why or how teachers implement these products. Is teacher use mandatory, 
voluntary, or discretionary? *Is implementation piece-meal, with teacher's 
picking and choosing from among various components oP* programs and only 
using them occasionally with subgroups of the pupils targeted for these 
materials? In some cases teachers' implementation will be pro forma , while 
others become intensely committed. These issues will be, elaborated on in 
latter chapters of this volume. 

We also paw in the previous chapter that the scope of product imple- 
. mentation is strongly related to other school outcomes. Thus, the question- 
arises as to how implementation can be increased. That is, what factors were 
related to teachers' inclinations to actually use the adopted product in 
their classrboms?^ We explored t,his question in two separate analyses. 
The first analysis presented in the previous section of this chapter, used 
only product characteristics as predictpr variables.- 

A second analysis of teachers' reported implementation behavior 
used a fuller r^nge of indicators of the RDU treatment' to identify factoid 
related to the inclination to implement. The purpose of adding additional 
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variables to the regression is to 'test bh§ possibility that-the strong 
relationship- between product characteristics and scope of implementation 
"washes out" when more powerful 'factors are entered. '(T^le other variables 
used m this regression will be - defined in greater detail in Chapters 6 
and 7.) The results of this stepwise regression are summarized in labile 5-3, 
and again show the importance of product characteristics, even when other 
elements of the RDU intervention wer« included as potential explanatory 
variables. Validated product,s seen to be of high quality resulted in 
higher levels of implementation. However, scope of implement atibri was also 
significantly enhanced by. contact -with external resaUrce persons: between* 
the local action team and the field agent, and with multiple soujrces of 
training in prt}duct use. Because the external assistance appears -to be of 
equal importance to the products, in this preliminary analysis, it is to this 
topic that we turn in the nextf chapter. 
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Table 5-3 

STANDARDIZED REGRESSION COEFFICIENTS FOR SCOPE OF IMPLEMENTATION 
REGRESSED ON .PRODUCT CHARACTERISTICS AND OTHER VARIABLES 

(N=60) 



Product Variables 

Product Quality ■ ^ .23* 

Difficulty of Implementation 

Field test/validation status ' .17 

Relative 'advantage 

Complexity 

External T.A. Variables 

Aqent/Principai Contact 
Agent Innovativeness • 

Agent Initiative ^ .20 
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Agent Time on' Site ^ .11 

Agent Political Perspective 
Agent Structural Perspective • 
Amount of Training .19* 
Variety of Training Sources 

Satisfaction with Agent ^ 

I nternal Process Variables 

Principal Influence 

Faculty Influence 

Breadth of Involvement in v> ^ 
Solution Selection 

Level of Effort 

Quality of Problem Solving 

Satisfaction with Problem- * 
Solying Process 
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Multiple -.49 
Adjusted Multiple R^ .43 



*P i .05 91 
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CHAPTER 6 



THE IMPACT OF ^RAINiNG AND TECHNICAL ASSISTANCE 



INTRODUCTION 

«. External assistance was provided to schools- in the R4D Utilization 
program by two types of people — genf&ralist field agents and more speciajized 
"trainers or expert consultants. In this chapter we explore the ways in which 
both fieFd agents and expert t»*ainers related to the participating school , 
l^ow the schools assessed these services, and their direct and indirect 
impacts upon the school. The specific questions *to be addressed include: 

• What kinds of help and assistance did RDU field agents 
and othe* consultants provide to local sites? 

• What styles or types of assistance were most effective, 
as judged by local sites? 

• What is the impact of technical assistance upon school 
improvement outcomes? Is impact direct or indirect? 

These questions must be viewed in a context of existing controversies 
about the importance and role of external pfoviders of technical assistance, 
both in the field of education and in other settings. Since the mid-sixties, 
some educators and others involved in /'technology transfer" have argued 
that the pace and quality of adapjbation being demanded of today's schools 
, require the deyelopnent of a role that is similar to that of the agricultural 
extension agent, v^o has contributed so significantly to the development and 
modernization of rural farm communities (CASEA, 1965). At tne same time, 
however, the distaste of educational practitioners for outside experts has 
been well documented by almost every observer of planned change activities in 
schools (Schmuck, 1968)* Some observers of schools have even suggested tNat 
the culture and structure of schools "may prevent them from making effective 
use of any systematic external assistance, such as that provided through an 
organisation development program (Derr, 1976). 

More recently, policy researchers studying federal demonstrations 
Of programs supporting local intervention have suggested that the ineffec- 
tiveness of "outside experts" occurs not simply because "experts" and "prac- 
titioners" view the world in different vays, .but because experts tend to 
want to impose their 'Own ideas upon a school or district, and ere t^herefore 
incapable of gaining the necessary commitment from staff to get them to view 
a new program as their own (Rosenblum and Louis, 1981$ Greenwood et al«, 
1975)* At the same time, however, there is growing evidence to suggest that 
local school people often do need help in carrying out a planned change 
program. Fot example, the most careful packaging and instructions for 
implementation which accompanied the PIPS (Project Information Package) 
demonstration program did not eliminate the need for some personal assistance 
in helping to design a local implementation effort (Stearns et al., 1977). 
Similarly, the study of an early effort to develop new field agent roles 
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m education revealed that genera list agents had significant impacts in 
stimulating interest m knowledge utilizaticm^ and at least some effects on 
actual usfe even with very smal4 investments'^ of time spent with a client 
(Sieber et al., 1972)* In addition, a recent study >»qf organization develop- 
ment activities' m local schools indicates that, while the presence of an 
active internal trainer and "change agent" is critical, the role of external 
expertise is very important (Miles et al., 1981). 

Overall, we know very little currently about why educational agents 
have an effect — either negative or positive. There are many volumes that 
advocate pertain strategies for producing change (for example, Zaltman and 
feuncan, 1977; Zaltman et al., 1978), but the empirical evidence has grown 
only modestly since Rogers «nd Shoempker^s (1972) and Havelock's ( 1969J 
massive reviews of thp literature. Thus, our approach in this analysis is a 
highly exploratory one. 

# 

TYPES or EXTERNAL ASSISTANCE .IN THE RDU PROGRAM 

While the RFP for the KDU program did not require a field agent 
role, each of the seven winning proposals included such a rotC, The role, 
partly by coincidence, but mostly as *a consequence of the program's focus on 
knowledge utilization and school improvement, had several common features 
across projects. 

\ First, m all the projects, field agents from outside the school 
system (known variously as "linking agents," •*generalists," "coordinators," 
and "facilitators") were expected to provide ih-person services to schools 
at the school site. The RDU program supported lOQ field agents during the 
course of the program. a ^ 

Second, m all cases, the field agents -were physically located 
outside an ROD project office, m a "host organization" that was geographi- 
cally closer to their client schools. The "host organizations" wer6 predomi- 
nately state-related intermediate service agencies. There were, however, 
some other agencies that housed agents. For example, the NEA project housed 
its "facilitators" m State Departments and state educat ign associat ions in 
eaeh state that it served. The NETWQRK/Consort lum* project used a variety 
of agencies, including a teacher center, a regional lab, and an LEA. 

Third, m all cases the field agents weie viewed as coordinators of 
the process assistanc e that schools would need if they were to choose to 
implement improved curriculum and staff development practices. Process 
Assistance typically involved, at mmunum, orienting school personnel to a 
rational"* prob lem-solvmg model that sites w'ere expected to use. In some 
cases, however, the field ag^^nts were expected either to participate m 
training, school staff, or to prov^ide the staff with substantial process 
consultations as they implemented a problem-solving fwodel, 

* Fourth, agents were not expected to take responsibility for finding 
exemplary programs for the c I lent schools to implement . This function 
was typically performed by specialists located felsewhere m the project 
structure. However, they were expected to provide schools with assistance 
m making choices f rom among alternative new pract ices, and to help them 
locate human resources that could assist the schools with implementation. 
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Finally, field agents in t*ie RDU program were all trained as educa- 
. tors, aryj almost aU-^d had some relatively recent experience working with 

school 'district-sj" either as independent consultants or as staff of a state 
p. education* association. They were, on the whole, much closer to the world of 

practice than to the world of research and development. 

Who Were the RDU Field Agents, and. What Did They Do? 

People became involved as field agents in the RDU program in a 
variety of ways. Some assumed the position by nature of their present^ 
jo^s — simply adding one rtHDre set of responsibilities to an already full 
complement of activities. Others were hired from the ranks of teachers and 
administrators to become full-time field agents — essentially leaving their 
ofd . responsibilities' behind. ' And for a few who were unemployed at .the time, 
the position was the first suitable job to become available*. For some, the 
field agent position offered ..the potential for individual challenge and 
•professional development, while for others the extra work involved reduced 
their enthusiasm. 

The field agents were highly ed^ih^^ted: of the 53 respondents to 
the first field ageht survey, * all but one had an advanced degree beyond the 
baccalaureate; ,70% achieved a master's degree, and 30% held a Ph.D. or 
C^.D. The field agent job, came at varying times in their careers. For some, 
this was their first "real" job after obtaining their most recent degree; for 
others this would be the last "formal" job prior to retirement. While these 
♦extremes did exist, the average age at the time of the first survey was 
41 — very much a mid-career stage in life-. The age of the field agents varied 
widely by project, from an average of 34 years in the Pennsylvania and 
NETWORK projects to an average of 47 years in the NEA project. Of the 
respondents who answei^ed all three , surveys, there were noore male (24) than 
female -(19) fiej^d ajgents. 

Since the seven projects all began at the same time — though some 
i.ere slower in hiring than others--there were no marked differences by 
project in the number of months of experience as an RDU field agent. At the 
same time of the first Purvey, 16 months was the average, length off time in 
^ the field agent position. It should be noted, however, that a number of the 
respondents came to this position from backgrounds that were quite relevant — 

' *The data sources described in detail in Chapter 3 are supplemented 

in this chapter with data obtained directly from fi^ld agents. The most 
important agCfTt data source was a three-wave mailed survey^ which was sent to 
a sample of 69 of the 100 field agents. The 69 agents represented the 
universe of. agents in six of the seven RDU projects, and a sample of 18 in 
the 'Michigan project. Fifty-three field agents responded to the first 
survey", which was sent out in May 1978 (a return rate 78%), with a 100% 
return rate from f6ur the seven projects. The somewhat lower response rates 
from the Michigan and NEA projects was not unexpected, given the very small 
part of these respondents* jobs represented by their participation in'the RDU 
program. 

The second and third sutve^ys , ' sent out in January .ynd May of 
1979, were completed only by those field agents who responded t6 the first 
survey. 
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for example, a few were associated with National Diffusion Network facili- 
tator projects or were consultants based in local school districts or inter- 
mediate service agencies. Seventy-five percent of the respondents to the 
first survey had had experience with other federally funded programs, 63% had 
had experience with other "linking" roles, and 35!o had had experience with 
R&O products or outcomes. 

Field Agent Roles 

There are many different perceptions of what educational field 
agents should do. The research and theoretical literature usually describes 
the field agent role in terms of the. problem solving/knowledge utilization ^ 
process. For example, Havelock (1973) has identified four change agent 
roles, labelled "catalyst," "solution giver," "process helper" and "resource 
linker." The field ag^jnt can serve as a catalyst by helping Jjchool district 
personnel to overcome their reluctance to change. He or she can then simply 
offer a solution, or gudde local staff through the stages of a logical 
problem-solving process. The agent's access to human, financial or other 
resources is also of great importance. Butler and Paisley (1978) also 
describe the roles of "process helper," "solution giver" and "resource 
finder," and Madey (1979) has most recently suggested three role categories: 
"facilitator," "resource finder," and "communicator." 

In our research we attempted to discover the extent to which the 
field agents in the RDU program perceived themselves as fitting into a 
fixed list of role' categories, chosen to reflect the roles described in the 
literature and our perceptions of actual variations in the RDU field agent 
role. The field agents were asked to assess the extent to which they had 
expected to perform certain aspects of the field agent role, and the extent 
to which they actually performed those roles. Responses to these questions 
for the 43 agents who responded to all*three surveys are summarized in Table 
6-1, with the potential roles listed in descending order of actual perfor- 
mance.* 

It is clear that the field agents perceived themselves primarily 
as resource persons and coordinators . Some of the activities that the 
field agents neither perceived as important nor actually performed were 
active involvement in program implementation, involvement in evaluation, and 
providing content specialist assistance. These activities are highly spec- 
ialized, and involve skills that many of the agents did not personally feel 
they had. 

For the most part, their actual role performance was consistent 
with their own expectations. There are, however, two exceptions: the . 
field agents felt that they should have been performirtg the role of an expert 
in assessing the match between innovations and problems to a greater extent 
than they were actually performing that role. Further, they performed the 
role of counselor or "hand-holder" to a greater extent than they expected. 

*See Louis and Kell, 1981, for fuller description of f^eld agent 
activities, including a sample weekly log and several case studies. 
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Table 6-1 

RANKS AND MEAN RATINGS OF FIELD AGENTS' EXPECTED AND 
AdTUAL EXTENT OF PERFORMANCE OF VARIOUS FIELD AGENT ROLES s 

(N = 43) 



- 


Field Ag^nt Roles 




Field Agents' 
Expectations 




Actual 
Performance 




• " Rank 


Me an* 


S.D. 


Rank 


Mean* 


S.D. 


a. 


Resource Person 


1 


4.5 


.7 


1 


4.2 


.-94 


b. 


Coordinator 


2 


4.3 


.9 


1 


4.2 


1.0 


c. 


Process Trainer 


3 




1.1 


3 


3.3 


1.1 


d. 


Observer/Histor lan 


6 


3.2 


.93 


3 


3.3 


1.1 


e. 


Counselor or "Hand-Hoider" 


6 


3.2 


1.1 


3 


3.3 


1.1 


f. 


Expert m Assessing tha^ 
Match between Innovations 
& Problems 


3 




.9 


* 

6 


3.0 


..95 


g- 


Conflict Resolver 


5 


3.3 


1.1 ' 


6 


3.Q 


1.1 


h. 


Basic skills. Career 
Education or Inservice 
Specialist 


8 


3.0 


1.2 


6 


3.0 . 


1.2 


1. 


Program Implement or 


10 


2.6 


1.2 


9 


2.6 


1.3 . 


J- 


Evaluator 


9 


2.8 


1.2 


10 


2.5 


1.1 



♦Response Scale ; 

5 = to a very great extent 

4 = to a great extent 

3 = to some extent 

2 = to a little extent 

1 = not at all 
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Field Agent Activities 

The above discussion of role definition focuses on the more global 
parameters of the roles field agents play. Yet, from the perspective 
of a job occupant the activities that make up the day-to-day cycle of events 
are in many ways more salient and more likely to stimulate positive or 
negative reactions than the more general role definitions. Based upon 
interviews with a sample of agents, a list of routine field agent activities 
was generated and included in a survey of agents. The RDU field agents were 
asked to rate the importance of each activity, and the amount of time spent 
on At. The results are shown in Table 6-2. On average, the field agents 
were spending the greatest amount of time in (1) meetings with small planning 
groups at the sites, (2) writing reports and filling out forms, (3) arrang- 
ing, designing, or conducting workshops, and (4) travelling from site to 
site. 

In general , there was little discrepancy between the amount of 
time the field agents were spending on various activities and the degree of 
importance they attached to these activities. There were , however , these 
notable exceptions : developing themselves professionally and reading mate- 
rials about R&D products were both thought pf as more than moderately impor- 
tant, ranking second and sixth, respectively, among the 16 possible activi- 
ties, and yet they consumed relatively little of the field agents' time. 
This is consistent with, the finding that the field agents felt they should be 
performing the role of an expert in assessing the match between innovations 
and problems to a greater extent than they were actually doing. The field 
agents appear to have taken seriously the notion of themselves as links to 
knowledge about R&D products or innovations, at the same time feeling some- 
what inadequate in the extent to which they performed this function and, 
perhaps, in the extent to which they currently had the knowledge and exper- 
tise to perform it well. 

There is also a discrepancy between the importance of, and the 
amount of time spent in, writing reports or filling out forms and travelling 
from site to site. That is, both these activities rank low in importance but 
high m the amount of time they consumed. Indeed, writing reports and 
filling out forms is the only activity which was ra^ed lower in importance 
{% - 2.1) than in the amount of time it consumed (x - 2.5). The conflict 
between "paper work" and "people work" was one that arose again and again in 
interviews and discussions with agents, who by far preferred their field and 
technical assistance roles to office work. 

Expert Technical Assistance and Training in the RDU Program 
« 

One of the key concepts underlying RDU was , the not loa oF- "net- 
working," which was often interpreted as the provision of timely organiza- 
tional resources to individual client schools. (For more discussion of 
networking in the RDU program, sue Louis and Rosenblum, 1981 . ) Overall , 
the provision of technical assistance and training from pe^ons Qther than 
the field agent was probably somewhat more limited than ^the^priginal de- 
sigrlers of the RDU program hed intended. For example, the original RFP did 
not mention the terms "linking agent" or "field agent," but spent consider- 
able time explicating the importance of inter-agency linkages that would be 
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Table 6-2 



RANKS AND MEm* RATINGS OF PERCEIVED IMPORTANCE AND 
\CTUAL AMOUNT OF iL.E SPENT ON VARIOUS FIELD AGENT ACTIVITIES 

(N = 43) 





Field Agent Activities 




Importance 






Amount of 
Time Spent 








Rank 


Mean* 


S.D. 


Rank 


Mean** 


S.J. 


a. 


Meetings with small planning 
groups at the sites 


1 


2.8 


1-5 


1 


2.5 


.6 


b. 


Writing reports/filling out 
forms 


11 


2.1 


.6 


1 


2.5 


.7 


c. 


Arranging, designing or 
conducting workshops 


3 


2.6 


.6 


3 


2.2 


.8 


d. 
e. 


Traveiinq from site to site 

Promoting or explaining the 
ROIJ program 


10 
4 


2.' 
2.5 


.8 
.6 


4 
5 


2.1 
2.0 


.7 
.6 


f. 


Working with individual 
admm ^'^tratora^ 


4 


2.5 


.7 


5 


2.0 


.8 


9- 


Organizing, preparing, and 
delivering materials 


6 


275 • 


.7 


5 


2.0 


.6 


h. 


General meetings with site 
staff 


6 


2.4 


.5 


5 


2.0 


.7 


1. 


Developing yourself profes- 
sionally y 


2 


2.7 


.5 


9 


1.9 


.7 


J- 


Meetings with RDU central \ 
project staff 


9 


2.3 


.5 


9 


1.9 


.7 


k. 


Reading materials about R&D 
products 


6 


2.4 


.7 


11 


1.7 


.6 


1 • 


Managing budgets 


11 


2.1 


.7 


12. 


1.6 


.7 


m. 


Designing, administering, 
and analyzing evaluation 
materials 


13 


, 2.0 


.7 


12 


1.6 


.7 


n. 


Observing teachers 


13 


2.0 


.7 


14 


1.5 


.7 


0. 


Working with individual 
teachers 


15 


1.8 


.8 


15 


1.3 


.6 


P- 


Working with parents or 
volunteers 


16 


1.6 


.7 


16 


1.0 


.3 



♦Response Scele: * *Re8ponse Scale; 

3 = very important 3 = a great (j .al of time 

2 = somewhat imr' "tant 2 = a ^noderate amount of time 

1 = of little or no importance 1 = little or no time 



nece8s0ry in order to provide appropriate services to schools (NIE, 1976). 
The notion of organizational linkages as a key feature of the intervention is 
even more explicit in clher agency documents. For example: 

The program hypothesizes that when internal capacity (of 
schools) IS insufficient to supply needed services, the 
involvement of one or more external organizations (or 
individuals) may be required. One agency .. .might not 
have the necessary capacity and resources to deliver all 
of the required services. Hence, linkages or arrange.iients 
between agencies need to be created to provide necessary 
resources. . .(Hut chins, 1976) 

The funded projects, on the other hand , tended to emphasize the 
role of a sinqle individual — a linking agent, field agent, facilitator, or 
"general ist" — whose responsibility it was to deliver or "broker" services to 
the school'. While the emphasis on the field agent, as opposed to linkage 
or "linking agencies," varied among the seven funded programs, the field 
agent played a dominant role from the beginning, and one which tended tt qrow 
in importance as the projects matured. Nevertheless, most of the St-i.ools 
involved with the program had at least some experience with project-sponsored 
training or technical assistance in addition to the services provided through 
the field agent. As will be seen later in this chapter, such training proved 
to be a very important* aspect of the services delivered and was strongly 
related to the achievement of some of the program impacts. 

These experiences were m^o^t pronounced in projects that had a formal 
design for providing training in problem-solving procedures to the sites. 
' Florida project, for example, provided two intensive training ppograms in 
n-solving and knowledge utilization skills to two or three representa- 
, of each of the schools that were active clients of its program (see 
uouis and Rosenblum, 1981, for more detail). Similarly, the schools involved 
ircPennsylvariia* s School Improvement Project were served by a "School Assis- 
tance Team," which consisted of the field agent, a representative from 
Research for Better Schools (who was to provide experiential training in 
probr^^^soiv ing processes), a representative from RISE, a non-profit informa- 
tion service agency, and a representative from the Learning Research and 
Development Center (LRDC) at the University of Pittsburgh, who could respond 
to Qny questions about basic curriculum issues. In other projects, however, 
the .approach to providing technical assistance, other than that from the 
field agent, was more ad Hoc. In the NETWORK project, for example, the field 
agent was considered to be the broker of all other services received by a 
client school. When the agent felt that the school needed technical assis- 
tance from another source, he or she would lecommend it. The agents also had 
their own budgets for providing such additional assistance. However, there 
were no project-wide standards about where or when assistance should occur 
although all schools tended to get at least some assistance and training 
related to the implementation of the selected product. 

Most of the sites involved in the project got some form of ''process 
training"^ related to improving their problem-solving practices. In our 
intensive sample of 90 sites, for example, approximately 55% were estimated 
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to have had ^ome identifiable training of this type. Most of this was 
provided by the field agent or other staff of the RDU project. In addition, 
contact with consultants who were to assist in ir.plement ing a planned 
change activity was prevalent. School survey data indicate that approxi- 
mately four out .of every tive schools received some training that was not 
provided directly *by the field agent or other project staff member as part 
of the program. 

In the vast majority cf cases thjts technical assistance or training 
was directly related to implementing a chosen product, rather than being 
part of a broader solution to the identified problem: only one out of every 
five of the schools in our smaller intensive sample had incorporated inser- 
vice training activities that were part of solving the jjroblem, but were not 
directly related to specific implementation issues and problems. 

The intensity of training for implementation varied enornrroirsly 
between schools and,- since in most cases the training was voluntary, even 
varied considerably within schools. The survey of teachers indicates that 
training experiences tended to be concentrated prior to implementation. 
Pre-implementat ion training of 25 hours or more occurred fo"r 32S5 of 
teachers, while similarly intensive experiences during the first implemen- 
tation year were reported by only 20%. Similarly, the percent report- 
ing eight or fewer hours of training was 42S5 before implementation and 5^% 
during the first year. 

The most frequent providers of training and technical, assistance 
related to implementation, other than the field agent, were the product 
developer and other staff members in the school district. In'^each case, 
approximately 2/3 of the teachers reported getting at least some training 
from people in these roles, ^hile half of the teachers reported receiving 
training from a consultant who was not the product developer, or a member 
of the RDU project or their own district, the incidence of this type of 
supplementary training for implementation was less frequent and occurred "to 
a very great extent" in only 19% of the cases, as opposed to 38% in the case 
of product developers.^ ' 

FIELD AGENT ASSISTANCE STRATEGIf S- AND CLIENT SATISFACTION 

One of the assumptions frequently discussed in the innovation 
literature is the resistance of schools to external providers of technical 
assistancer The generic problem of bridging the gap between knowledge 
producers and potential users by developing "linking roles" has been exten- 
sively treated by Havelock (1969). On the other hand, empirical studies of 
field agents who adopted "linking" roles have indicated that they too may 
find access to school districts and schools problematic (Louis and Sieber, 
1979; Louis and Kell, 1981). In this section we explore the ways in which 
the field agents* approach to technical assistance and change affects tea- 
chers* and principals^ assessments of their usefulness.* ^ 



♦Equivalent data are- not available for consultants of other types. 
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Agent Perspectives and Strategies 

As we and othera hava observed (Louis, Kell, Chabotar and Sieber, 
1981; House, 1981; Deal and Nutt, 1980; Sieber, 1972), the ways in which 
individuals who have responsibilities as change agents view the change 
process will have major implications for the strategies that they select to 
carry out their role, Sieber points out: 

As one srsriS the tactics that are pursued in bringing 
about focused change in educational systems. . .one is 
struck by both the wide variety of approaches and by 
the high degree of confidence displayed by proponents 
of each different technique. . .the many approaches... 
can be subsumed under three basic strategies, each of 
which IS rooted in a particular image. 

(Sieber, 1972, pp. 362-363) 

Sieber goes on to define the strategies associated with each set of personal 
-^images, bo^ he is ambiguous about whether a change agent's strategies arise 
from the agent's views of the school char\ge process, or whether the agent 
develops* a coherent set of images to correspcnd with his or her preferred 
strategy. Our own position is that the belief system, or personal^ imagery, 
that -individuals bring to the ta sk of organizational change will condition 
their willingness to s e llB c t v a r Fo u8~ st T fft^g i es to support or stimulate 
cnange. T4Hut:, in some sense, the personal imagery of the agent is a surro- 
gate* way ^f le«»:ing at the overall "game plan" which change agents or field 
agents are likely to adopt. 

Through focused but unstructured interviews ,with field age^nts and 
school personnel yie have identified three different perspect ives ^bout 
what is cf primary importance in accounting for the outcomes of any activity 
or events in schools. The first of these images is the structural perspec- 
tiv e. This perspective emphasizes the social structure of the school as a 
formal organization and the ways in which this must be altered in order 
to allow ch«nge to occur. ' A second dominant perspective is the individual 
incentives perspective , which errphasizes individual needs, incent ives ,' 
and disincentives for change. The final significant imagery is t^he political 
perspective , which emphasizes the need to understand-*-and manipulate — the 
power structure of the sfchool in order to implement change programs. 

Lach of these images clearly suggests strategies for change. thus, 
for, example, we would expect the agent who believes in the individual incen- 
tives perspective to spend more time working through individual acceptance 
and participation in decision making than one who believes in the political 
perspective. Similarly, a structural approach might emphasize developing a 
plan for how a new curriculum package would affect the, job definitions and 
informal social structure of the school (e.g., teactier ytime for socializing 
and exchanging information) while a political orientalion might attempt to 
look for the interest groups that would be the biggest barriers to carrying 
Out the impletnentation plan. 

The major question to be addressed in this section is whether the 
field agents' images of the change process are related to their perceptions 
of client success, and clients' assessments of field iigents* performance, 
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Measures of Perspectives 6n Change ^ 




The three perspectives were measured by asking the agents to com- 
plete a set of six forced-choice questions. Each question paired a s^^e- 
ment reflecting one of the perspectives with a statement refj.ecting another 
perspective. Each time the field agent made a choice, he or she was given a 
score of ot\e for the orientaLjon that they chose. Thus, the possible range 

for each orientation was between zero and four. « 

- » 

In addition to these direct measures of perspectives on change, 
we also "use a measure of field agent innovativeness as a surrogate fot the 
degree to which the agent is likely to choose highly visible, novel, and 
creative strategies for initiating - change, versus low-keyed, facilitative 
strategies. The measure of innovativeness involved forced-choice selection 
between pairs of adjectives describing the respondent's behavior. Four 
innovative characteristics ^independent , flexible, original, and self-reliant) 
were paired with four conventional characteristics (depe'^id^gible, cooperative, 
industrious, stable). An innovativeness score is obtained by adding thb 
number of times an innovative adjective is selected over a conventional 
adjective. (See Price, 1972, for more- information about this measure.) 
Support for viewing innovativeness in this way-mdy be found in its correla- 
tion with other strategy measures. It is positively correlated with both 
a political perspective (r = .22) and an individual incentives perspective 
(r = .30).* 

Client Assessments of Agent Perforjuance ^ 

Four measures of client evaluations of the agentjs performance are 
used. Two of these are direct: tRachers* ratings of the effectiveness of 
their agent on 13 dimensions, and principals* ratings on the'^me scale. In 
addition to" this ♦d-.rect assessment of field agent performance, both teachers 
and pri/icipals wen^ adked for their assessments of the process which the 
agents had led them through. The measurement of. these four outcomes is 
discussed in detail in Appendix 6. Finally, clients were also asked for 
gldbal assessment of agent services in a battery which also included items 
ftfr other providers of technical assistance. 

Analysis and Findings: Agents'* Perspective and Strategies ^ 

and fclient Satisfaction 
« ■ * 

Perhaps the most outstanding finding regaraing the different perspec- , 
tives on change is a simple descriptive one: field agents overwhelmingly 
.eschew a "political" strategy for creating change, one which emphasizes power 
groups both as facilitators and, potential blockp (Table 6-3). By far the 
greatest consensus among them is a preference for an individualized ^l^roach 
to change, which stresses wprkihg through individual motivations, concerns 
and reactions. Not only is th^ mean preference for this modus operandi 
highest, but the variance among agents is relatively low. 



' ♦Note that these correlation CQef ficients ar,e computed based on 
48^ linker respondents. The correlation matrix at the end of this chapter,^ 
on the other hand, involved linking all field agent sd^res to each site 
with which they worked. Thus, they are not comparable. 
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Table 6-3 



MEANS AND STANDARD DEVIATIONS FOR 
THREE FIELD AGENT PERSPECTIVES ON CHANGE 

(Ns47) 



Field Agent 

P^spective Mean* Standard Deviation 



Political Perapective . .88 . 1.21 

Social Syatem Perapectiye 2.34 ^ .97 

Individual Incentives 2.71 * .78 
Perapective 



y 



♦Scale rangea from 



The preference for the individual incentives approach is not surpris- 
ing. . As Deal and Nutt (1980) have noted, it is the popular approach for 
most educators; 

» 

Many administrators find individual personalities — although 
complex and volatile— easier to' understand than the dynamics 
of complex systems such as schools and school districts. 
They often overemphasize the cohesiveness and rationality 
of the system and their own ability to control. • .the acti- 
vity and sentiments of others. 

The emphasis upon the individual incentives strategy for creating change has 
been critiqued elsewhere (see Deal and Nutt, 1980; and Louis, Kell, Chabotar 
and Sieber,, 1981) where* it has been observed that this^ strategy ^nores the 
importance of the formal organizational structure of schools. It should be 
noted, however, that ^the field agents typically supplemented the preferred 
individual approach with a large dose of structural strategy, which involves 
understanding roles, divisions of labor^ and rational organizational planning* 
processes.* 

Does the lack of sympathy toward a political strategy represent'* an 
irrbalance in the "bag of tricks" that field agents use to create an appro- 
priate environment for change? As we shall, see below, the answer is a mitced 
on«, but, on ther^whole, we can conclude that agents might profit from a 
greater recognition of how power works in formal organizations — and how the 
change agent can-plan to use the power system to facilitate participation and 
rational planning. * 

Overall we Yound that clients tended to be very satisfied with the 
services provided by the field agents. When teachers were asked to rate 
their overall satisfaction with various sources of personal assistance 
provided to them during the project, the field agents were rated as most 
helpful by the teachers and principals who- resnjg!x ! bj bcx oev?xtk Prpduct 
developers were rated as considerably less helpful by both teachej's and 
principals (Table 6-4). 

^ Simple correlations between teacher and principal satisfaction with 
the process and with the field agents, ^nd the measures of agent perspectives 
and strategies are shown in Table 6-5. This table reveals that an innovative 
orientation on the part of the agent ha3 a negative impact on principal and 
teacher satisfaction n^h the RDU process, and their assessments of the 
hejpft/lness of the agent . 

Because the finding that .innovativeness has siich strong negative 
relationships with site perceptions of agents is not necessarily consistent 
with all of the literature about desirable personality characteristics of 
change agents, i^ is useful to speculate a bit further about the meaning of 
this relationship. FiY-st, it may be^ noted that thd concept of innovative- 
ness includes some characteri§tic8 that are thought to be positively asso- 
ciated with effective j::hange agents— flexibility, and the ability to be 
self-reliant. On the other hand, it* 'also incorporates, other attributes 
that may be less compatible with "linking agentry"— namely, originality and 
asking questions. Boundary-spanning persons are often expected to be both 



liable 6-4 

SATISFACTION OF TEAtHERS AND PRINCIPALS WITH VARIOUS 
PROVIDERS OF TECHNICAL ASSISTANCE AND TRAINING 



Teachers ^ 

Field Agent (N=461) 
Other RDU Staff (N=304) 
Product Developers (N=362) 
Other Ccnsultanta (N=307) 

Principals 

Field Agent (N:il34) 
Other ROU Staff (N=118) 
Produbt Developers (N=118) 
Other Consultants (N=118) 



iot Satisfied 




Very Satisfied 


Total 


1 


2 


3 


4 


5 


f 


5% 


• 10 


% 

25 


29. 


31 


J 

s^OOS 


855 


13 


35 


• 29 


15 


lOOS 


11% 


11 


28- 


27 


24 


lOOS 


9% 


11 


34 


27.' 


19 


130K 




9 


17 


' 32 


40 


lOOK 


2% 


7 


29 


, 40 


23 


100% 


3S 


8 


31 


36 


22 


100% 


•5% 


13 


28 


31 


25 


100% 
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• ' ' Table 6-5 

PEARSON CORRELATION OF FIELD AGENT PERSPECTIVES AND 
/ STRATEGIES AND CLIENT SATISFACTION " 
(Significant Correlations Only) 



FIELD AGENT 
PtRSPECTIVES 
AND STRATEGIES 



TEACHER 
- SATISFACTION 
WITH PROCESS 



TEACHER 
SATISFACTION 
WITH AGENT 



PRINCIPAL 
SATISFACTION 
WITH PROCESS 



PRINCIPAL 
SATISFACTION 
WITH AGENT 



Exten,t of 
Agent Inno- 
vativeness 



-.28* 



-.33* 



Agent 

Political . 
Perspective 



-.28* 



.24* 



.26* 



Agent 
Individual 
' Incent ives 
Perspective 



Agent 

Structural 
p Perspective 



-.19* 



^♦Significant at the .05 level. 
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innovative and able to fade into the background — an expectation tnat is 
probably unreasonable. 

The need for low^keyed^ dependable, cooperative, and industrious 
behavior is clear, not only from the statistical findings presented here, 
but also from the interviews with |3oth field agents and clients. The field 
agents emphasized the low-keyed, non-initiating part of their job as a key 
to success ("let them think they did it — you're just the support"). Clients 
praised the agents for their ability to chair meetings and orc^nize support, 
b^ut not > for their imagination or originality. The need for non- innovative 
personality characteristics may be particularly critical in the case of 
the external change agents, where, as outsiders, their legitimacy to intro- 
duce novel or original ideas is suspect. However, we suspect- that even for 
the inside change agent, self-reliance or originality is les^ important ire 
creating^ a mandate for d^^ange than cooper at iveness and stability. 

Also shown in' Table 6-5 are the findings that field agent perspec- 
ti^ves on change have scattered relationships with the outcome variabJ^es. 
The more pronounced the individual incentives orientation of the agent/ the 
more satisfied teachers are with the process (r = .20). This is probably a 
result of the fact that an individual incentives model is more teachec-cen- 
tered than the other two. A political perspective on the part of agents, on 
the other hand, has a negative relationship with teacher satisfaction with 
the process (-.28), but is positively associated with principal satisfaction 
with the agent (r = .26) and the process (r = .24). Since the political 
orientation is associated with strategies to use the power structure to 
achieve change, an agent .holding such views would be very likely to spend 
more time with aSministrators than teachers, thus causing these results (see 
Louis and Sieber, 1979, for additional data to support this). 

Overall, th^ findings suggest that the perspectives and strategies 
that the agent chooses. to bring to the relationships with clients do have 
an impact upon clients* assessments. However, the only clear pattern that 
emerges from the analysis is that agents who are highly innovative in their 
orientation are less likely to be successful. The scattered quality of the 
relationships between other independent and dependent measures indicates that 
different perspectives may work well in some settings, and with, some role 
groups, and less well in others. Thus, we are led to the tentative conclu- 
sion that there is no one strategy thrat is particularly effective (or in- 
effective) in schools. Rather, the relationship between agent and client is 
probably much more complicated and dependent upon local- features. Based on 
our case ^^erials, however, we also beXieve that the sparse findings emerg- 
ing from this 'Analysis may be more a reflection of the difficulty of captur- 
ing these elusive relationships between agents and their clients than of 
their actual .signif icance; * 

In addition to the simple correlational analysis, canonical corre- 
lations were computed to examine the total effect of the perspectives and 
innov at iveness as a group upon client satisfaction. The first canonical 
correlation of .47 was significant at the .005 level. Correlation coeffi- 
cients for the first canonical correlations are presented in Table 6-6. 
Th-is tabl? confirms that innov at iveness and'^he political orientation are 
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Table 6-6 

CANONICAL CORRELATION COEFFICIENTS BETWEEN CLIENT SATISFACTION 
AND AGENT PERSPECTIVES - » 

(N= 38) 



Group I 




Group II 




Client Satisfaction 


Corr. 


Agent Perspectives 


Corr. 


Teacher Sat. W/Aqent 


.78 


In»Jividual ^ 




'Prin, Sat». W/Aqent 


.71 


Political 


.62 


Teacher Sat. W/Proces9 


.07 


Structural 


-.37 


Prin. "^at. W/Process 


.75- 


Innovativeness 


-.72 


Canonical correlation: 


.^7, 


signifitant at the .^005 


level 
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the most powerful variables in predicting clie^^t satisfaction. Specifically, 
to repeat earlier findings, innovativeness relates negatively to all four 
measures of client satisfaction — teacher and principal satisfaction with the 
process and with the agent; the political perspective is also related posi- 
tively to these measures, leaving out teacher satisfaction with the agent, 

THE IMPACT OF EXTERNAL TECHNICAL ASSISTANCE ON SCHOOL 
IMPROVEMENT OUTCOMES 

The logic and literature support ing the anticipated impact of field 
agents and consultants on local school improvement activities were discussed 
in Chapters 1 and 2, However, there is still controversy over whether 
external technical assistance has significant g^osit ive effects on local 
change efforts. Current 'research to date supports a full gamut of answers 
to that question, ranging from assertions that external consultants actually 
harm local change activities (Greenwood et al,^ 1975; Berman and Pauley,* 
1975) , to those who find strong positive effects (Louis and Sieber, 1979; 
rEmricI: and Peterson, 1976; Miles et al. , 1978), An important question that 
we seek to answer is, therefore: what is the overall impact of external 
change agent activities upon school improvement processes? 

In addition, there is a second controversy that we seek to address, 
which focuses on the degree which generalist field agents or more 
specialized trainers are important, and how each type may contribute to 
innovation at the school l&vel. Thus, for example, Zaltman and Duncan 
(1977) argue that strong specialization and expertise are extremely important 
criteria for any external change agent and that low levels of expertise will 
undermine credibility and impact. Other studies, however, have indicated 
that generalists and specialists may have similar impacts upon knowledge 
utilization among educators (Louis, 1975). Still others imply that genera- 
list agents may have more impact in the pre-implementation stages, while 
spe^cialists have more impact in the later stages. Thus, a question that will 
be addressed is: what is the relative impact of field agent, assistance an d 
specialized consultant assistance ? 

Measures of External Technical Assistance 

Seven measures of external technical assistance were used in th.; 
analysis. Two of these refer to the "expert" training and consulting 
assistance provided as part of the project: total amount of training re- 
ceived and variety of providers ,* 

Five measures relating to f leld agent activity were used: field 
agent initiative in providing services , field ag^nt intensity of services, 
the amount of co ntact between the field agent and the principal , and the 

*Total alnount of training was computed by adding the percentage of 
teachers reporting more than 25 hours of training before implementation to 
the^ percent reporting more than 25 hours after implementation. Variety of 
provider s was computed by adding the number of different types of providers 
from a list including district specialists, other district staff members, 
product developers, field agents, other project staff, and other consultants. 
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measure of field agent mnovativeness and political orientation that emerged 
as significant m th^ previous section.* , 

• Analysis and Findings: Agent & Training Impacts • ^ * 

An examination of the correlations betwe^ six scfiool improve- 
^ ment outcomes and the seven measures of external te^^hnical pssista«te indi- 
! cates that three of, the external assistance variables seem to have relatively 
little impact (Table 6-7). While^ as we saw 'previously , an . innovative or 
political orientation may have negative impacts upon clifent assessments of 
agent performance, it is not significantly correlated with any of the 
dependent variables. In addition,* principal estimate^f their contact with 
the field agents does not affect the actual outcomes of the change process 
(with the exception of organizat^o^ change). 

Two additional field agent variables do correlate strongly with 
s&veral of the outcome variables, however. Field agent initiative and 
mteni^ity a^e correlated at the .01 level of significance with the overall 
index of organizational improvement, with program incorporation, and wxth the 
sc(y)e and magnitude of ch&nqB-^rrXbe school.. 

fhe correlations between the two variables measuring the anourit 
of training and the variety of training sources show even stronger relation- 
ships with the dependent variables, however. In botn cases, there is a 
significant correlation with each of the outcome measures, and these are 
equal to or exceed the magnitude of the correlations between field agent 
variables and outcomes. 

Table 6-8 presents the results of ^the regression of the six dependent 
variables upon the seven external technical assistance mea,sures. The results 
indicate that the external hunan assistance provided to schools can have 
major impacts upon the degree to which knowledge is used and school improve- 
ment occurs. Technical assistance and training act ivities^have particularly 
potent impacts ^on overall organizational change and program incorporation. 



♦Field agent initiative was measured by adding together 13 vari- 
ables from the consolidated coding form: ^ 10 o'f these variables were measures 
of the degree to which agents delivered more intensive types of services 
(such as providing training, or helping to diagnose the problem). An addi- 
tional measure examined afield agent influence over the school's decisions 
at four stages m the process, and was measured similarly to the internal 
site actor's influence discussed m Chapter 7. Another measure estimated 
the importance of the agent to the schpol's activities. The last measure 
tapped the amount of initiative used by the^agent m his/her attempts to 
influence the schools. Intensity of services is composed of variables 
indicating the proportion of local team meetings attended by the agent^ 
m the four stages of the problem-solving process. These measures are frcJn 
the consolidated coding fbrm. The. measure of agent->principal contact is 
a single indicator from the principal survey,, which asked the respondent 
to estimate the amount of contact .on a four-point scale ranging from "a 
lot" to "none." 
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Table 6-7 

PEARSON CORRELATIONS BETWEEN MEASURES OF 
SCHOOL IMPROVEMENT AND EXTERNAL TECHNICAL ASSISTANCE VARIABLES 

(N=75) 



School 

iMprovetnent 

Outcone 



External Assistance Variables 



Field Agent Contact Bet. Agent Total Variety of 

Field Ag^t Intensity Field Agent Innovative- PoJitical Amount of trtintng 
Initiative of Services & Principal' ness Orientation Training Sources 



Organizational 
Inpact 

iCi Incorporation 

P'^oduct 
Inc-rporation 

Scope of 
Implementation 

Pfoblem 
Salved 

Personal 
Impacts 



.33*» 

03 

.34" 

.22 
-.01 



.33»» 



,19 



-.10 



.26* 

.10 

.22 

.12 

.20 

.11 



.09 



-.07 



-.06 



.11 



-.15 



-.04 ' 



.02 



.05 



.02 



-.08 



-.00 



-.07 



.44»» 

.27* 

.31»» 

.40*» 

.33** 

.34*» 



.44*» 
.30»» 
.59»» 
.41»» 
.38»* 
.26* 



ERJC 



• p < .05 
•* p I .01 
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Table 6-8 



STANDARDIZED STEPWISE REGRESSION (BETA)* COEFFICIENTS 
FOR THE RELATIONSHIP OF EXTERNAL ASSISTANCE AND SIX 
MEASURES OF SCHOOL OUTCOMES 
(N// 76) 



External 

Assistance 

Variables 



School Outcome 



Organizational Product ' Process Problem Scope of „ Personal 

Impacts ^corporation Incorporation Solved Implementation Impacts 



Field Agent (FA) 
Initiative 


ff 




.24* 




f^fK Time on Site 


.19 


.23** . J 




1 D 

lo 


FA Political Perspective 






-.13 




FA Structural Perspective 






• 




FA Innovat iveness 










FA Contact with Principals 


.16* 


.13 






Amount of Training 


.33** 


.10 .17 .24* 


.28** 


.26** 


Vajfiety of Training Sources 


.25* 


.43** .22* .31** 


.19 


.21* 


Multiple 


.40 


.43 .14 .21 


.46 




Adjusted Multiple 


.36 


.40 .10 .17 


.41 




+ Beta Coefficients are presented only for those variables which contributed to 
The selection process was stopped when additional variables failed to increase 
more; the order of entry was unforced. 


2 ' \ 
the reported multiple R . 

the Multiple R"^ by IX or 
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* P < .05 
** P I .01 



125 



where 36Si and 4(K of t^he variance are explained respectively. Only process 
incorporation arxJ personal impacts are poorly explained by the level of huwwn 
assistance. ) 



Three variables stand out as being most important, and of these, 
one IS related to field agent behaviors, and two are related to training. 
The ^ aim^unt of training received by the site staff prior to iinplementatton 
and after implementation has a strong positive effect, and this impact 
13 augmented by having training provided by a variety of different types of 
people . The tyne that the field cTgent spends with local site ^committees 
or "problems-solving teams" is predictive of several dependent measures. 

Our site visits revealed that much of the importance of the agents 
can be attributed to the role they played on site both in stimulating 
committee members to stay active and ^each decisions and in providing logis- 
tical support to ensure that the meetings were scheduled regularly, that 
suggestions for consultants were obtained, etc. Thus, the actual presence of 
the agent on site was important. 

For example, m one school that we visited, the staff members re- 
ferred to the field agent as "our superego — she gave us the kick that we 
needed; she'd tell us what we had to do..." In another case, the field agent 
HB3 viewed as instrumental in raising disagreements between staff members 
that would have otherwise gone unresolved: 

(The field agent) was sensitive to the- fact that con- 
sensus did not exist on any definition of the problem, 
but that group members were willing to claim they agreed 
in order to avoid conflict ^. .For three long meetings the 
(group) grappled with problem identification. Whenever 
Hartwell detected differences of opinion, she would state 
what she thought was the source of the difficulty and in- 
vite discussion. . .demonstrations of her genuine interest 
in the group helped her to establish warm personal bonds* 
with •^he teachers at Jefferson. (Kraqs, 1981) 

Only spending a great deal of time with members of the school allowed the 
agent to play a role such as that described above. 

In many cases, staff members did not explicitly recognize the signi- 
ficance of the field agent to their activities. This may be in part a 
consequence of the general reluctance of schools to admit to influence 
by external actors (Louis et al., 1980:286-88; Miles, 1981), and in part 
because some of the agents who spent a great deal of time and energy working 
with clients had, nevertheless, a sufficiently low-key style that their 
contributions were very unobtrusive: 

Milton had no desire to act like a pushy expert ^nd was 
content to keep a low profile, offering only an occasional 
comment* If the group had no need for his skills as 
facilitator, they did look to him as a resource person... 
(Desmond, Louis and Murphy, 1981: 194) 



114 
1 Op 



The Relative Impact of Field Agent Assistance 
and Specialized Traj>ning Assistance 

There is a tendency, revealed both by the quantitative and qualita- 
tive data, for the two types of external human assistance to have somewhat 
different impacts on the' site. General i&ts and field agents have their 
greatest impacts m stimulating . the school staff to define their problems 
more broadly,' and to think more ambitiously about what might be done to solve 
t^V thus producing a ^hange progrrMn of greater magnitude. The multiple 
^ J£S££' example, is the only regression equation m Table 6*8 

that IS dominated by field agent variables* * Field agents and cFients in 
•♦auccesafal" achoois often agreed that a major effect of the problem' defini- 
tion and solution selection pfNOcess, m which field agent activity tended* to 
be greatest, was to increase both the breadth of* interest among potential 
uaera and to result m a "solution" that required more change on the part of 
the teacher. 

Did thid mean that agents were, ii^ fact, "product pushers," who 
advocated for mqre major changes? . The answer to this is, typically not. 
Case-materials reveal that most agents bdnt oyer backwards to refrain from 
influencing th^ actual product that was chosen: 

^ J[The agent) did not, like the ECRI program, and pri- 

vately remarked that she would not want her own child 
m^an ECRI class. Even. so, she as^uned a professional 
neutrality and pushed for a fair consideration of it. 
(Kraus, 1981:204) 

♦ * 

In another case where the agent wished to wor'^ through local administrators 
m a more centralized decision-making process, the school-based team ^ver- 
theless had a great deal of influence: 

(The agent and two district administrators) then began ^ 
to look for information on promising career education 
materi^s. . .after reviewing the ones that were unfamil- 
iar [tcrthem] they selected ten for revj^w by the teachers ■ 
whb would be implementing the products. • .During the per- 
lod from April to July the team met... to discuss and sift 
through the products (and make a, selection).. (Halpern, 
1981) 

In th%: above case^ the agent had a direct influence on scope by presenting 
product^ that were highly varied content and materials, but which were 
all based on the notion of infusing career educatiorr through major portions^ 
of the curriculum*. It is important to note, however, that the impact of 
field agents on outcomes appears to occur bot*h direptly (py achieving broader 
scope of imolementation and, to a lesser degree/ incorporation of the new* 
program or practice); and, to an even greater%extent, indirectly (by in- 
fluencing the problem- solving process m school).* For example, both the 



♦The degree to which the activities of external change agents predict 
the quality of problem solving m schools 4s discussed m greater detail m 
f Chapter 9. / . • * 
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amount of field agent time s^'pent m the school, and the level of field agent 
init/iative and activity are strongly correlated with the level of effort 
(measured m number of person days) expended by the team members and other 
participants m the school (r : .35 and .43, respectively, both signi- 
ficant at the .01 level). Similarly, the quality of the problem-solvmg 
process used m the school la significantly correlated with bojth theue 
measures of field agent characteristics (r = .23 and .27, respectively, 
both significant at the .05 level). Finally, there are significant posit^ive 
correlations between these measures and the level of broadly-based faculty 
participation m decision making (r = .23 and .26, Both significant at the 
.05 level). 

These f#indings are not surprising based on obseivatiotle o f ^eld 
agents m action."" For the most part, agents tended to dominate th^ deliver.y 
of services m the early stages of each school's problem-solving process. 
They were of critical importance as generali^t sd^porters of the. schools' 
efforts to develop a better definition of their, problem and a structuj^e for 
decision making and m assisting them' to develop criteria and a process for 
determining what available products should be reviewed. As we have noted 
eatlier m this chap^[^, the role of generalise as "process" helper was the 
one that they^ themselves felt most comfortable with. Thus, i^^ appears that 
to a very large degree t%e more active field agents' were achieving what most 
of them hoped for7-to improve the quality of the problem-solving process, 
to increase the salience of problem-solving activities in the routine of the 
school, arid to make problem-solving and decision making more participatory. 

Once a product had been selected for implementation, on the other 
hand, the school staff'3 interest and center of activity turned to the speci- 
fic content of the innovation and the need for focused training for implemen- 
tation, rather than tQ decision-making skills. Most field agents had little 
expectation that they would play a strong role m implementation (as noted 
earlier m Table 6-1), nor did they see themselves as content specialists. 

' While most RDU projects had a clear differentiation of roles for 
process assistance (i.e., the field agent role) and for product information 
assistance (i.e., the knowledge-base specialists), the actual job definitions 
for the field agent were largely ambiguous. Some viewed themselves as 
resource persons and coordinators and expended a heavy effort m develop- 
ing their roles for the early stages of the problem-solving process. They 
received little preparation or orientation from either the project headquar- 
ters or their own host organ izations f^or their appropriate rol-e and activi-. 
ties m the pre-implemcritstion and implementation phases. Many appeared to 
lack the instructional des.ign or supervisory skills that would be most useful 
during implementation, or had difficulty applying the relevant skills which 
they might have had. Furthermore, the "ambiguity of their* positi^on wajs 
augmented when faced with the need to define a role which would be both 
compatible with and different from that carrieo out by tbe technical experts 
who were- brought m to assist m implementation activities. 

For example, one particularly forthright agent expressed great 
anxiety after her schools had chosen products, saying that she could not 
really figure out what her role should be during the year and a half they 
would be implementing the new program. She, and other agents, cautiously 
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tried out new roles, including designing and helping to carry out evaluation 
programs, dr brokering (and occasionally compensating for/)^ other external 
resources of a more technical nature. Others simply becam^ involved in new 
activities, either in different schools or with the same clifeAts. 

• , . • 

The relative importance of technical expertise in implementation and 
the\ctual determination of the degree of change and improvement in a school 
is revealed by Table, 6-8. While field agent activities hadVlittle impact on 
measures of overall school improvement, the degree to which the problem was 
solved, and process incorporation, both the amount of draining and>variety in 
the types of training provided was important to each of these outcome vai'i- 
ables. The contribution of the two is orbicularly impressive in the case of 
overall school improvement and the degree to ^which the problem was solved . 
In the cars of these outcome v a r i ab 1 es, both field agent and training var^- 
ables contribute one percent or more to the overall adjusted multiple R • 
However, the relative contributions are distinctively different, with the 
•*standardized regreasiori coefficients being much greater for the variables 
reflecting formal training activities. In summery, the message of Table 6-8 
(as well as the simple correlations in Table\6-7) is clear: effective 
training is a loy to effective school j.mprovement] programs. 

Site visit data lead us to the obser/ation that the statistical 
relationships shown here obscure somethingj>r a "chicken-and-egg" problem. 
Heavy local interest in external and inte/fial training activities tended to 
occur in those 'schools that were already well on their way through a success- 
ful school improvement program. While interviews with teachers suggest that 
training activities were imgprtant to sustain commitment and ensure appro- 
priat|» implementation of the product, effective training typically did not 
stimulate most of the impressive school improvement outcomes, but rather 
reinforced them in significant ways. 

The notion that effective training is an outcome as well as a cause 
(Sf a school improvement process is suggested by the correlation of training 
variables with other indicators of the "quality" of the process. This 
measure reflects early activities as well as those occurring in preparation 
for or during implementation (the periods during which training typically 
took place and to which our training measures refer). The level of effort 
devoted to problem solving in the school and the quality of the problem- 
solvii^ process are both correlated with the 'variety of trainers (r = .29, 
sig. = .05) and (r = .34, sig. .= .01). Similarly, schools w^th high levels 
♦of faculty influence tended to have more training (.26, sig. = .05) and more 
different trainers (.27, sig. = .05). Finally, broad-based involvement 
in solution selection, ^^ich preceded the choice of trainers involved in 
preparing for implementation, was strongly correlated with a variety of 
trdmers (.29, sig. = .05). 

In fact, qualitative data and tjie observations of field agents 
tend to suggest that in mapy cases the most effective schools shifted from 
depending on their field agent as a major source of/ external stimulus to 
depending on a single or set of technically expert training sources during 
implementation. While the shift was rarely complete, it was often- substan- 
tial in cases where appropriate and excellent trainers were available. 
Schools which were lucky enough to choose produces accompanied by relatively 



inexpensive and enthusiastic ueveloper/trainers tended to became the most , 
dbmmitted to thbir new courjje. Particularly important, according to many of 
the sites that were visited, wasi a trainer who later, returned several times 
after the first pre-implementation training session, or who was available for 
telephone consultation. Another- very effective mode in a few sites was 
"turnkey training," where a limited number of stefff members or administrator/ 
specialists received intensive training, usually at the developer's own 
school, and then retijrned as local experts. Obviously the staff who went 
benefited more than those who stayed behind, Mjut an on-site^ presence was 
helpful. - , . 

The difference between the impacts of field agents and other "train- 
ers" is nowhere more evident that in the case of gtaff development outcomes. 
If the simple correlations are examined (Tab^e 6-7), it appears that field 
agent behaviors have%io impact upon staff development outcomes, while train- 
ing variables are quite strongly correlated. , In the multiple regression, 
this finding is further augmented. The standardized regression coefficients 
for variables contributing at least one peroent to the adjjusteil multiple' 
R include both training variable5^, which have a positive relationship with 
staff development, and field agent time sp^nt with the team, which has a 
negative relationship (significant at the .07 level). 

A number of issues are raised by this finding. First, it implies 
that while schools can effectively implernent externally developed products 
without having to "reinvent the wheel," and that an active field agent can 
effectively help to assist in this process, , high levels of invblvement from 
an external faqilitatpr may inhibit the developftient of strong internal 
capacities at the individual level. We did not observe many instances irj 
which it was apparent that a field agent's presence inhibited the emergence 
of internal leadership awng the teachers, but it is possible that* more 
staff development occurreo^in those sites which had field agents who did not 
•or could not take on all of the facilitating responsibilities associated with 
carrying out the process. 

Trainers, on the other hand, whose contact with the schools tended 
to be much more episodic and limited m duration, as ^ell as more focused and 
specialized, were apparently able to facilitate both personal growth outcomes 
and program implementation outcomes. We believe that the explanation for 
this finding lies m the fact that during the pre-implementation and imple- 
mentation activities — in which the trainers were most' substantially involved" — 
most teachers were able to focus on actual problems they faced in the class- 
room. The expeits were typically screened and selected to provide training 
targeted to a specific need for information or skills associated with the 
program being implemented. .Thus, it provided teachers with skills and ideas 
that they could use right away. It seems that teachers are likely to benefit 
most when they focus their energies on needs that are central to their daily 
work life in the classroom (see also Huberman, 1981). This finding does "not 
obviate the importance of broad involvement in the problem-solving activities 
(which' are discussed m the following chapter). What it does suggest is that 
it is crucial to get to the "bottom line" of teacher needs if effective staff 
development is to occur. 
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Having found one instance of a possibly negative impact of involving 
external field agents in a local school improvement process, it is reasonable 
to c)3k whether other negative impacts may be found. Among the possible 
negative effects that may be postulated are that, external field agents will: 
(1) simply supplant activities that would have been carried out equally well 
by scl^iool staff; *(2) attempt to inappropriately monopolize the training and 
service provision role with client schools and possibly avoid qalling in 
technical experts when they are actually needed; or (3) tend as outsiders to 
make mistakes ir>i judgerner|t which may impede the acceptance of new solutions 
to local problems'. We h^ve already addressed the first issue, having indi- 
cated that fi^ld agent involvement was positively related to indicators of 
local level of effort and initiative. The question of whether field agents 
will tend to try to dominate a local problem-solving process even in areas 
where they are not competent to provide training cannot be supported, for we 
find that both agent time and agent initiative and activity are positively 
.correlated with the variety of training received (.28, sig. = .05), suggest- 
ing that, if anything, they are more likely to propose the use of alternative 
human resources. Finally, our qualitative data indicate that most agents 
were extremely careful to work very slowly with the local schools until they 
understood the "culture" of their environment quite well. The, data suggest 
that the external agents were quite effective in stimulating a "bottom up" 
approach to probleiji solving, rather than the "top down" approach that is more 
typical when the district office, calls in an "Expert" to solve problem. 
In school after school, teachers expressed the belief that the process they 
were participating in was qualitatively different and significantly more 
participatory than previous committees on which they had worked. Indeed, in 
some instances, it was the first time teachers fait that they had the oppor- 
tunity to select a new program to be used in their school, as opposed to 
ratifying a. selection macje by the principal or central office. In summary, 
there is no evidence to suggest that there was any significant pattern of 
external agents producing negative effects on outcomes in the local district. 

SUMMARY 

The provision ot external technical assistairice was a major 'xiuiponent 
of the RDU intervention strategy. In this chapter the severs^ ypes of 
external assistartce provided were discussed, the impact of the peispec^ives 
and strategies applied by the external agents were related to client satis- 
faction with external assistance, and the impacts of external assistance on 
schotjl improvement outconfes were examined. Based on the above data, the 
following conclusions may be drawn: 

• Most of the schools involved in the program received 
two q6ite different types of external technical 
assistance. Field agents, who were at least part- 
time employees of the projects, typically provided 
more ongoing and sustained assistance to the 
schools. Field agents (and their clients) typically 
viewed their appropriate role as resource persons 
and coordinators, who were there to facilitate the 
schools' progress. ^ 
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• In additioRi most schools also received focused 
training from substantive experts Mho assisted the 
sites. m program implementation. The prevalence of 
this role differentiation m technical assistance is 
a dramatic testimony to the impacl^ of Havelock's 
"linkage" model on practice, smce-it was neither 

^ *^ required nor even hinted at in,the original program 
RFP, 

• Field agents were perceived by teachers iind principals 
as more helpful than the providers of specialized 
assistance. Overall, clients preferred sustained 

ipport rather than episodic training events, 
iients were particularly satisfied ^ith field agents • 
who- displayed high levels of initiative m providing 
assistance, and tho9e who spent a lot of time on 
site. On the other hand, field agents who were' 
self-reportedly more innovative were not perceived 
as helpful, 

^^^^^ 

• fioth field agents and specialized trainers had 
significant positive impacts on school improvement 
outcomes and, m general, the more ^^sistance received 
by both typea of technical assistance providers, the 
greater the benefits. However, despite the field 
agents* popularity with clients^ the amount and 
variety of training from "experts" had a greater 
impaction school improvement , 

A major implicatibn of these findings is that it is important not to 
confuse measures of clienb satisfaction with technicai* assistance with the 
actual impacts of that assistance. In addition, we should also note that the 
impacts of technical assistance strategies on long-range school improver/ient 
outcomes are T\ot equivalent to 'the full range of benefits that schools may 
obtain from receiving technical assistance. For example, as we have argued, 
field agent activities may have indirect effects upon school problem-solving 
behaviors. In addition| other studies have^ shown that field agents have 
significant impacts m both recruiting clients, and sustaining their involve- 
ment through adoption (Louis and Sieber, 1979; Fmrick, 1977). We believe, 
that our findings t,end to support the notion that different agent roles may 
be appropriate at different stages m the problem-solving process, as has 
been argued by Crandall (197^), and this important^ topic is raised agaip m 
Chapter 9. On the other hand, we should not ignore the alternative explana- 
tion that the value of trainers and training m producing long-tenn impacts, 
as opposed to generalists and facilitators m producing more proxirnal out- 
comes, IS m part, confounded by the fac> that few of the RDU technical 
assistance providers were^ able to conceivfe of their roles^ as encompassing 
both general ist and expert assistance strategies. In ot^er words, pre-adop- 
tion and post-adoption technical assistmce needs may be different, but xt is 
certainly possible that one individual could fulfill both, at least to a 
greater extent than occurred m the RDU program. 
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CHAPTER 7 



IMPACTS OF THE INTERNAL PROBLEM-SOLVING P^tDCtSS* 
INTRODUCTION 

As mentioned in Chapter 2, the R&D Utilization program waa focuaed 
not only on the alleviation of a apecific problem in each site achooly but 
alao on the lasting improvement of each achool'a problem-solving capability. 
In this sense, . the RDU program waa engaged in organization development and 
capacity buiTding, aa well aa ,in knowledge diaseminat ion. Each project 
included in its design a number of atepa which each aite waa expected to 
accompliah as part of a rational problem-aolving/knowledqe utilization model . 
In addition^ each project required or encouraged broad-baaed participation in 
the sitea' problem-aolviYig procedurea. 

Although apecific procedurea varied acroaa projecta, there were 
certjBin key characteristics present in each project 'a problem-Qolving model* 
The common character iatica are: (1) thorough analyaia and prioritization of 
achool needs or problem before searching for achool improvement atrategiea; 
(2) a aearch outaide .the local achool ayatem for aaaistance and information, 
eapecially on alternative aolutiona to the identified problemsi (3) systematic 
examination of the alternat ivea, according to explicit criteria; and (4) a 
preference for aolutiona that had been field teated and empirically vali- 
dated.** ' In add^ioHf the projecta generally required aites to develop 
comprehensive ppeU'implementation plana or propoeala, which then had to be., 
approved by project^^|taf f. Broad-baaed participation in the problem-aolving 
proceaa waa encouraged through the formation of local problem-aoIving teama, 
known aa "aite teams," "deciaion-making groups," "local action teama," 
anrf^"taak forcea" in the varioua projects. Most projects insisted that both 
teachers and adminiatratora be repreaented on the teams. Several projecta 
also encouraged input from the faculty aa a whcle, through aurveya, polla, or 
faculty meetings. 

Within the outlines of' a project 'a problem-aolvxng model there waa 
substantial room for variation. Indeed, not all aitea adhered cloaely to 
their projecta' suggested procedures. An exairiple waa a middle achool whose 
"decision-making ^roup" consisted of four teachers, a counaelor, the principal, 
and the assistant auperintendent for curriculimi. The group diacuased problems 
and aelected a program during a two-day workshop which waa alao attended by 
their field agent. (There' waa no input from the faculty aa a whole.). The 
problems identified by the group were very general, diffuse, and not priori- 
tized; they alao akipped the procedure for specifying prpgram selection 
criteria. Additional information on the products waa obtained through 
several long-distance phone calla during the two-day workshop. A training 
session held for the entire faculty waa a poor introduction to implementation, 
since the planning had not taken into account the differencea between the 
adopting achool and the original implementation site. As a reault, moat 
of the achool'a teachera became^ diagruntled and later withdrew from the 
program. 



*Thi8 chapter was written by Diane Kell. 
♦*Most projects required sites to select 8olutior>a from the projects' 
own knowledge baaes. Since the projects specialized in solutions for 
problema in readino, math, and/or career education, there waa a limit on 
the typea of problema that could be addreaaed through the RDU program.^ 
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At the other extreme was an elementary school whicH took three 
years to identify and analyze the school's instructional -problems, ^select 
a program to alleviate those problems, and prepare for implert^ent at ion. 
The process began with a one-hour orientation session for the entire. faculty. 
Members of an external consultant group later visited the school to conduct 
individual interviews with the teachers and to observe therr classrooms. 
Next, a preliminary analysis ot school problems (based on the visat) was 
shared with a "local action team," including six teachers, a ^counselor, 
the assistant superintendent for curriculum, and the principal. During 
the following months, eight full days of team meetings were devoted to 
further specification of student and program needs. Between meetings, a 
great deal of effort was expended collecting and analyzing data through 
teacher interviews, student testing, and other means. 

One year after ^en^^mg the program, the school completed its 
problem statement. The team then spent two full days developing the criteria 
for program selection and scre^enipg the choices presented to them by the 
external consultant group. Six options were presented to the faculty during 
an all-day meeting, and the faculty's choice of a reading management system 
was later confirmed by the pfoblem-solvmg team. Several team rneetmgs 
were then devoted to developing* an implementation plan. Th§ local action 
team next participated m a full week of inte.nsive mservice m the adoption 
of the reading management system; this was followed by a one-day session for 
the entire faculty, held two yedrs after the school's entry into the RDU 
program. Another year was then devoted to modifying the reading management 
system to meet local site needs ,and conditions. The entire' faculty was 
involved m this process through grade-level groups .headed by members of the 
local action team. When the program was finally implemented, the whole 
faculty felt proud of their effort, m spite of feeling that they could have 
accomplished as much m less time. 

A final example illustrates the range of site experiences m the RDU 
program. At this school all major decisions related to the problem-solving 
process were made by the faculty as a whole during regularly scheduled 
faculty meetings. However, a key role was played by three individuals—the 
principal, a central office resource teacher, and one of the school's .first 
grade teachers — who together acted as the "school facilitators." The. facili- 
tators attended two state-widp training sessions to learn how to lead the 
faculty through the process of defining a problem and selecting and imple- 
menting a sojlution; they were assisted by the project's field agent, who was 
very actively involved throughout the process. The first act ivity with 
faculty was a brainstorming session to identify problems with the school's 
instructional program. The results were then summarized by the school 
facilitators and distributed to each teacher Tor prioritization. Over the 
next few months the facilitators met several times to work on a problem 
statement and a request for information on available^ products. The drafts of 
these documents were reviewed by key faculty members and submitted six months 
after the school's entry into the project. When 'the product options were 
received from the project, the facilitators met several times to screen the 
choices and select three for presentation to the faculty. The faculty then 
met m grade-level groups to discuss the options and rank them m order of 
preference. The faculty made its final decision on product selection one 
year after the school's entry into the project. Over the summer there were 
several days of planning for implementation, involving school and district 
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administrators, as Mell as teachers and consultants* The school impljemented 
some aspects of the new program the next fall but received intensive in- 
service m all components of the program on a monthly basis. Withm two 
yearSy the changes m the school were profound and involved all aspects of 
schoo^ operations, from teaching methods to school management. 

■ lAi 1 three of the above examples, teachers played a key role 
in decision maktog. However, there were'^Jso sites where, whether o; not e 
problem-solving team was formally established, the decisions were made by 
school or district administrators. These sites represent yet another vari- 
ation ^from the. idealized problem-solving models of the RDU projects. 

^ *In this chapter, we examine the importance of. both rationality 
and broa^-based participation m local school improvement processes^ relative 
to several outcomes which have already been described: organizatiohal 
impacts on the school; incorporation of the selected program; incorporation 
of the problem-solving process; the extent to which teachers report the 
problem solved; scope of implementation; and personal impacts on teachers. 
More generally, this chapter describes local school problem-solving behavior 
— including the roles of district administrators, principals, and teachers — 
and examines the effects of internal roles on school ^provement outcomes. 
t v> 
The rollAiying section describes the variables and measures used to 
assess the internal* problem-solving .process. The overall impacts of the 
process, as indicated by ste|)wise multiple regressions on the outcome 
measures,' are described in the section dealing with "Overall Impacts of the 
Internal Process on Outcomes." In "Participation and Influence in Decision 
Making" and "Other/ Character ist ics of the Problem-Solving Process" each 
component of the proc^ess is described and analyzed m greater detail. 

VARIABLES AND MEASURES 

kost of our data on the internal problem-sobrlng process come from 
our own site visits and from site case materials ^i^vided by the projects. 
These qualitative data were encoded by members the study team, using 
the consolidated coding technique described in Chapter 3. 

One group of variables describes the influence that actors in various 
role^ groups within the school s stem harf over major decisions in the school 
improvement process. Separate . tems were developed for each role group 
during e^h stage of the process-; problem identification; solution selec- 
tion; planning for implementation; and ongoing implementation monituring, 
evaluation, or planning. Coders were asked to select the response category 
which described ttie mos^ influential member of the role group, rather than 
the average degree of influence of members of the group.* The role groups 
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♦The responsjB categories for each item were: 
None or ve;"y kittle : Had little or no input into decisions, and 

littjie.or no influence. 

Some; May have had considerable input into decisions but was not 

a strong influence. ' 

A great deal : Strongly influenced the decisions; may hav^ made the 

final decisions alone. 
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considered m the analyses were the superintendent or assistant superintend 
i^eit, other central office staff, the principal's or assistant principalst and 
teachers.* We also measured the influence of the faculty as a whole and the 
, problem-solvinq team . In both these cases, coders were asked to assess the 
degree of influence exercised by the group as a unit rather than as indivi- 
duals. A single measure of mflu^ce across problem-solvinn stages was 
computed for each group.** 

A second group of variables describes breadth of involvement during 
each of the problem-solving stages. These variables were constructed from 
items on the conso)idated coding form assessing he extent of involvement 
of the most actively involved merrbers of each rolf group during each stage 
of tne process. Since the scores were added across role groups to measure 
breadth of involvement, this group of vari^les lakes into consideration not 
only the number of role groups participating m the process, but also the 
extent of their involvement. The role groups included m the measurement of 
this group of variables were the sujierinte^dent or assistant superintendent, 
ot^ :»r central office staff, the principals or assistant principals, teachers,^ 
and other school-level staff. 



A third group of variables describes the level of effort , m person- 
days of local staff lime, for each stage of the process. Total level of 
effort across stages t.as computed by adding the items.*** 

Finally, there are two groups of variables which describe the 

"quality" of the local process. Thp first assesses cbngruence with "sound*' 

group decision making practices, snd the second measures congruence with a 

"rational" problem-solving model. The criteria for assessing quality of 

group decision making and quality of the problem-solving process were adapted 

from criteria developed by Sam Sieber on the basis of a review of the litera- 

iture. The criteria and instructions to coders are reproduced in Appendix 
/ f ^ # # # * 



*In addition, the principals* assessments of their own involvement in 
the schools* problem-solving activities were taken from the principal survey. 
The scale is composed of four items corresponding to the four stages of the 
process. The variable was included in the analyses to test the hypothesis 
that an active, though not necessarily directive, principal is important to 
the success of school-improvement efforts. However, since the variable was 
not powerful ir the analyses, it will not be discussed further. 

**Variablf^s spanning the stages were computed by adding the responses"^ 
for each stage, excluding the implerentation stgfae, which had a larger pro- 
portion of missing cases, f 

***ror each stage, coders could choose from the following response 
categories : 

Low: Less than 10 pecson-days 

Medium : 10 to 30 person-days 

High ; Over jQ person-days 
♦♦♦♦Other researchers and practitioners may disagree with some of the 
criteria; after all, there is no single set of established nprms for group 
decision^makmg or problem-solving quality. However, since the criteria were 
drawn from the school-improvement literature and from the RDU projects* own 
problem-solving models, we believe they are an adequate measure of the extent 
CO which the local RDU processes were consistent with current models for 
rational, participatory school improvement, and thus they are adequate for 
determining the strength of the relationship between adherence to these models 
and success m a sehool-imr^ovement effort. 
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OVERALL IMPACTS OF THE INTERNAL PROCESS ON OUTCu. :S 



As shown In Table 7-1 , only one of our quantitative measures of 
school Improvement outcomes, organizational impacts, can be predicted very 
well with the internal process variables* This corresponds to our analysis 
of case data, which suggests that many sites arrived at "successful" school 
improvement outcomes via a wide variety of locally-designed routes, despite 
the projects^* attempts to encourage a particular problem-solving appSDach. In 
some schools, centralized decision making by the superintendent or princi- 
pal was highly effective; IjT^others, a decentralized, staff devejlopment 
^proach worked well* Nevertheless, our statistical analysis does Inaicate a 
modest level of predictive power for internal process variables, partiixularly 
for the overall organizational change outcome, but also for the process 
incorporation outcome, which is not well explained by other aspects orythe 
program intervention (see Chapters 5 and 6).» \ 

Four variables contribute tc organizational impacts: the quality 
of the problem-solving process; overall faculty influence on the process; 
breadth of involvem^t in solution selection; and brepdth of involvement 
in implementation (R = .34). Process incorporation (R = .15) is achieved 
through breadth of involvement in solution selection and implementation 
(though the beta coefficient on the latter variable is not significant) 
and through lower levels of influence from central office staff. In general, 
most of the predictive power of the internal process across all six outcomes 
is attributable to three variables: overall faculty influence on the pro- 
cess; breadth of involvement in solution selection; and breadth of involve- 
ment in implementation. Thus, the most important part of the RDU problem- 
solving approach appears to be its emphasis on participatory decision making. 

One of the surprises of the multiple regression analyses is the 
fact that principal influence^ is not a powerful explanatory factor, even 
though it is correlated with ^veral outcomes. Our analyris of narrative 
data on the sites indicates that^K-Wf^rrany of the : nst successful schools, 
principals facilitated the process of planned change, but preferred to let 
the process be teacher dominated. Thus, while not totally passive, the 
principals in successful schools did not always receive high scores for 
influence. This does not mean that they were ineffective leaders, but 
merely that they chose a nondirective leadership style. This point ip 
discussed further in the following section, and in Chapter 9. 

A second surprise is that the process does not predict level of 
staff development benefits, or personal Impacts, reported at a school. 
Based upon both theory and at least some of our site visits, we would have 
predicted that staff development benefits would have been more strongly 
a?sociat«d with process variables such as level of effort and faculty influ- 
ence. However, staff development outcomes, at least as they are aggregated 



^Each multiple regression analysis was performed using the same set 
of independent variables, all of which are listed in Table 7-1. Most 
of these variables were included in the analyses because of high correla- 
tions with three or more outcomes; however, superintendent influence and 
other central staff influence were included for their theoretical impor- 
tance. Team influence was included in earlier regression analyses, but 
did not ent^r the equations. 
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Table 7-1 

STEPWISE REGRESSION (Beta) COEFFICIENTS 
FOR RaATIONSHIPS BETWEEN INTERNAL PROCESS VARIABLES 
AND SIX SITE-'LEVEL OUTCOMES 



Site-Level Ootcomes 



Internal Proceaa 
Variable 



Organizational 
Impacts 
(nr90) 



Product 
Incorporation 
(n=90) 



Process 
Incorporation 
((n=76) 



Problem 

Solved 

(n=76) 



Scope of 
Implementation 

(n=90) 



Personal 
Impacts 

(n=76) 



I:; 



L-evel of Effort 

Quality of Problem-Solving 
Process 

Faculty Influence on 
Proc'ess ' 

Principal Influence on 
Process 

Superintendent Influence 
on ^rpcess 

Other Central Staff 
Influence on Process 

Breadth of Involvement 
in Solution Selection 

Breadth of Involvement 
in Implementation 



Multiple 

2 

Adjusted R 



.11* 



.IV 



.24*» 



.23* 



.13 



.29* 



,38 
.34 



.15 

.12 



-.13* 
.24* 
.20 



.20 
.15 



.2C»» 



-.20 



.15 
,11 



.23* 



-.15 



.31»» 



.16 
.12 



.12» 



.05 
.02 
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to the school level, are largely a function of the amount of training re- 
ceived by staff members (see Chapter 6). Further analyses were conducted 
using the individual teacher-lavel data to compare the personal impacts 
reported by team members and non-team members — all of whom were targeted 
users of the product. These investigations showed that, on average, staff 
members who were on the team during at least three of the four problem- 
solving stages derilied substantially greater personal benefits than those who 
were not on the team. For example, responses concerning the extent to which 
they had learned about curriculum development averaged 2.4 for teachers on 
the team compared 'with 1.7 for Jteachers not on the team (the response scale 
included 0 as the l6w pojnt and 4 as the high point). Similar results were 
obtained for other*" categories of personal benefits, as shown m Figure 7-1. 
These findings suggest that the problem-solving process can be a source of 
staff development , but primarily for those who are directly involved m the 
planning and decis*ion-making activities. 

PARTICIPATION AND INFLUENCE IN DECISION MAKING 

There are many accounts of the lack of successful planned change 
m schools. Typically, teachers develop lesson plans, learning objectives, 
and teaching stistegies on their own, with little interaction or influence 
from other teachers or administrators. Meanwhile, decisions about school- 
wide change are made by administrators, who then try to impose their decisions 
on teachers. 'The RDU approach is different m that it relies upon collabora- 
tion between teachers and administrators and the development of consensus 
about needs and solutions. 

A problem-solving team representing both teachers and administrators 
was formed m four out of five RDU sites, and m the majority of cases these 
teams exerted a great deal , of influence on the problem-solving process. Not 
surprisingly, however, the influence of ^individual teachers surpassed that of 
the team, as shown m Table 7-2. This reflects the strong influence of 
individual team members during problem identification and especially solution 
selection, as well as some decentralization of control during planning for 
implementation. Once solutions were chosen and minimal training given, 
teachers m many schools were once again left on their own to implement the 
solutions as they saw fit. In one out of three sites, ther^ was no »team at 
all, or it wis functionally nonexistent, by the time the solution was imple- 
mented. ' 

In post sites, the faculty as a whole had very little, if any, 
influence; however, the percentage of cases m which the faculty as a whole 
had at least some influence (e.g,, 43% during problem identification and 3855 
durinc; solution selection) was higher than m most organized change efforts. 
Real faculty influence was often cited by local participants as one of the 
keVs. features distinguishing RDU from more typical innovative activities 
m their schools. As discussed above and elaborated later m this section, 
the extent of faculty influence was a strong predictor of successful site 
outcomes. 

"^here was no clear trend in the influence of principals over decisions 
m the problem-solving process. On the one hand, a substantial proportion 
of principals (47?o during problem identification, down to 36% during implement 
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FIGURE 7-1 
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TABLE 7-2 

*LEVEL,OF IHFLUeNCE OVER MAJOR DECISIONS, 
BY RXE GROUP AND STAGE OF THE PROCESS 



Percent of Cases 



Level of Influence 



Teaa 



No te«i 

None or very little 
Some 

A great deal 



Problew 
Identification 



8 
10 

23 
60 

(nser^ 



Solution 
Selection 



13 
8 

20 
39 

T5Bg" 



Planning for 
Implementation 



17 
12 

22 
49 



Implementation 



27 
12 
19 
42 



(n=85) 



(n=77) 



(n=67) 



Teacher (s) 

,None or very little 
Some 

A gr^at deal 



17? 

27 

57 



100% 
(n=90) 



9% 

24 
67 

100% 

(n=87) 



9% 
31 
60 
lOOS 

(n=75) 



6% 
35 
59 



100% 
(nte69) 



Faculty as a whole 

None or very little 
Some 

A great deal 



58% 

27 

16 

(n=90) 



. 62% 

21 

17 
"TSDJT 
(n=85) 



71% 
23 

(nr78) 



57% 

31 

13 

(n=72) 



Principal /Asst. Principal 
None or very little 
Some 

A great deal 



26% 

26 

47 



31% 

27 

42 

TUBT 



30% 
29 

41 

T3Cr 



35% 

29 

36 



Superintendent/Asst . Supt. 
None or very little 
Some 

A great deai 



68% 

17 

15 

(n=88) 



74% 

14 

13 ' 

(nz87) 



79% 

10 

10 

(n=77) 



82% 
14 

(n571) 



Other central office staff 

None or very Uttle 
Some 

A great deal 



38% 

24 

38 

HRJoT 

(ns79) 



43% 
29 

28 

100% 

(n=75) 



53% 
15 

32 

100% 

(n:66) 



63S 

19 
19 



100% 
(n=59) 
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tat ion) exwrted a great deal of influence. On the other hand, there were 
many principals (26S during problem identification, up to 35% during imple«- 
mentation) who influenced the process hardly at all. The principal's role in 
the process is discussed further below. 

The large majority of superintendents or assistant superintendents 
(68S during problem identification, increasing to 82% during implementation) 
had little or no influence over decisions related to the school improvement 
^ pr«gects. It should be nnted,* moreover, that most ot the local projects 
'iJfere school-based rather th^n district -wide. , Other central office staff were 
more fikely to take part m the school improvement decx»ir6ns, though they had 
less influence than school-baaed administrators or staff. 

Principal influence, teacher influence, and faculty influence are 
positively related to one another, as well as to team influence, as shown 
m Table 7-3. However, there is a tendency, though not statistically 
significant, for the influence of the superintendent and other central office 
staff to be negatively related to the influence of school-bassv staff and 
administrators. This suggests two interpretations; first, that central 
administrators are more likely to seek involvement m the typical "top-down" 
model of organized change, rather than supporting change initiated or con- 
trolled from below; and, second, that participation by central administrators 
tends to perpetuate the "top-down" model. 

The importance of broad participation m the local problem-solving 
process has already been noted. The relationship between breadth of involve- 
ment and site-level outcomes is strong at each stage of the process, with the 
^exception of problem identification. It should be remembered that breadth 
of irtvo'lvement reflects not just the involvement of the faculty and the 
principal withm the implementing school, but also involvement on the part of 
the superintendent, ceqtral office specialists, and other relevant actors. 
For example, a high score on breadth of involvement m implementation typi- 
cally represented a district m which the central office tbol^ at least some 
interest m monitoring the , implenentation process, m providing support, and 
m spreading the new practice to other schools m the district, but did not 
dominate the process. 

Relationships between each of our measures of site success and the 
influence of actors m the different role groups by stage of the process are 
shown in Table 7-4. The influence of an individual teacher, or teachers, is 
shown to be relatively unimportant, while the influence of the faculty as a 
whole IS very strongly related to site success, especially in terms of 
organizational impacts. Apparently, it is not enough for teachers to be 
represented m the df cision-makmg process; rather, the process has to 
involve the entire facjlty, at least during critical decision points. 

The most important stages for broad faculty participation, judging 
^ from the correlations, are solution selection and ongoing implementation 
monitoring, evaluation or planning. Based on our site visits, giving the 
faculty as a whole the chance to participate m selecting the solution not 
only ensures that the solution will be acceptable to the majority of poten- 
tial implementors, but also helps increase the faculty's commitment to or 
sense of ownership of the innovation. Participation of ti>e faculty as a 
whole m ongoing implementation monitoring, evaluation, or planning helps 
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Ttble 7-3 

PEARSON CORRELATION COCmCIENTS 
rOR RELATIONSHIPS AMONG THE LEVELS OF INTLUENCE 
rOR DlfFERENT ROLE GROUPS 



InfluefMre of 
Tea* 

TeacberCs) 
Faculty as a Whole 
Principal/Aast. Princioal 
Super iniendent/Aaat. Supt, 
Other Central Office Staff 



Team 
• 

.09 
^.04 
.03 



Teacher(8) 



.34*» 
.24* 

-.15 

-.U 



Faculty 
as a Whole 



.31»» 

.20 
.10 



Principal/ 
Aaat. principal 



.09 
.07 



Superintendent/ 
Aaat. Supt. 



Other 
Central Office 
Staff 



.11 



P < 



= .05 
P i .01 



145 



146 



TABLE 7^ 

• PtARSON CORRELATION COEFFICIENTS 
FOR RELATIONSHIPS BETWEEN SIX SITE-LEVEL OUTCOMES 
ANO THE LEVELS OT INFLUENCE OF DIFFERENT ROLE GROUPS 
BY STAGE or THE PROCESS 



6lt«-L«v«l Qutco«e~ 



Role Group 


Organizational 


Product 


Proceaa 


Problem 


Scope of 


Personal 


by Proc«M StSQt 


lapacta 


Incorporation 


Incorporation 


Solved 


Inpleoientation 


Inpacts 


Jmm ' 














Problvn Identification 


.12 


.03 


.05 ' 


.02 \ 


-.06 


-.03 


Solution Selection 


.22* 


AX 


.02 


.06 


.02 


.10 


Plannin9 for I«pl. 


.23» 


.25» 


.01 


.20 


.06 


.04 


lBplM»ntation 


.13 


.09 


.09 


.00 


-.03 


-.07 


Total 


.28* 


.17 


' ^5 


.11 


.07 


.03 


TeacharCa) 






% 








Probia* Identification 


.IB 


.09 


.12 


.04 


-.04 


-.01 


Solution Stltction 


.11 . 


-.08 


.01 


-.02 


.04 


-.00 


Planning for iapl. 


. .20 


.10 


.06 


-.02 


'-.01 


-.00 


laplaaantation 


.13 


.21 


.02 


-.03 


.02 


.10 


Total 


.26* 


.27* 


.14 


.09 


.10 


-.03 



Faculty aa a llhole 



Problea Identification 


.25* 


-.01 


.09 


.1? 


.02 


.12 


Solution Selection 


.45»» 


.17 


.28* 


.37** 


.21* 


.35»* 


Planning for lapl. 


.35*» 


.20 


.04 


/.20 


.18 


.14 


lap lament at ion 


.48** 


.38^ 


.26* 




.39»» 


.20 


Total 


.50»* 


.30** 


.28»» 


^.35»* 


.22* 


.23» 


rrincipal/Aaat. Principal 














Probltm Identification 


.19 


-.00 


.13 


.'06 


* .08 


.09 


Solution Selection 


.39»* 


.16 


.19 


.25* 


.27»» 


.10 


Planning for lapl. 


.35*^ 


.27» 


-.03 


.24* 


• .19 


.09 


liipleaentation 


.40*» 


.25» 


.35»» 


.21 » 


.21 


.13 


&jperint«ndent/Aeat. Supt. 














Problem Identification 


-.04 


-.06 


-.00 


-.12 


-.17 


.14 


Solution Selection 


.06 


-.03 


-.07 


-.05 


• .01 


.06 


Planning for Impl. 


-.01 


-.12 


.26* 


-.14 


-.06 


.04 


Implementation 


.07 


-.04 


-.09 


-.07 


-.08 


.13 


Total 


-.05 


-.18 


.07 


-.23»» 


-.10 


-.02 



Other Central Office Staff 

Problem Identification .06 

Solution Selection .09 

Planning for Impl. .14 

Implementation .Q7 

Total .07 



• P < 



.05 



.02 
.10 
.13 
.04 
.04 



.10 
.19" 
.14 
-13 
.12 



.02 
.05 
.14 
.12 
.03 



.12 
.16 
.20 
.09 
.13 



.05 
.02 
.11 
.22 
.06 



P % .01 



147 



, 132 
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to reinforce individual use of the innovation through peer pressure and 
support, and also helps to minijnize ad hoc adaptations at the individual 
level. ^^^^ ^ 

A team is important to the extent that it facilitates breadth of 
participation and, above all, decision ^makiii^ by the faculty as a whole. 
In addition, a tean effort helps to systematize the prbblem-solvinq process, 
leading to higher levels of problem-sqlving quality, or rationality, a factor 
directly related to site success. The correlations between team influence 
and these other,' very important aspects' of the internal process are as 
follows: breadth of involvement in solution selection (r = .28, sig. .01); 
breadth of involvement in planning for implementation (r = .38, sig. .01); 
breadth of involvement in implementation (r = .34, sig. .01); influence of 
the faculty as a whole (r = .2f8, sig. .01); and quality of the problem-^ 
solving process (r = .46, siq. .01). In other words, team participation in 
decision making helps to accomplish school improvement objectives by spread- 
ing the ownership of decisions \(i .e. , atake-holding) ; by eliciting my^tiple 
perspectives, insights, and expertise, particularly from those closest to the 
• problem (i.e., teachers); by dampening the effects of a prion assumptions or 
pet theories of .-individuals or cliques; and by focusing attention on compar- 
ability and coordination across grade le^gis and classrooms.* 

Nevertheless, as L>hown in Table 7-4, the influence of the team 
itself IS unimportant, except perhaps in relation to our measure of overall 
organizational change.. For this reason, it. is„imp^Drtant to make a distinc- 
tion between three types of teams, all of which w'ere present in the RDU 
program: facilitating t^ams; decision-making teams; and implementing teams. 
Some characteristi^cs of these teams are ahown in Table 7-5. In some sites, 
one team served in all three roles; however, in other sites, the roles 
were divided among two or more teams, or among the team, the faculty, and 
various individuals. 

By our definition, -a facilitating team was one that initiated meet- 
ings of a larger decision-making group, planned the agendas for these meet- 
ings, helped to spark the enthusiasm of the group, structured and facilitated 
the group process^ collected and presented the necessary data fdr decision 
making, followed up on details between meetings, sought administrative 
^support and cooperation, and served as the primary contact between th» school 
and extern€\l consultants or information resources. In some sites, the 
facilitating team also screened the initial options for action and wrote the 
first drafts of problem statements, search, requests, and implementation 
proposals or^ plans. Facilitating teams were formally organized in some 
sites,, but in most sites the functions of facilitating the group process were 
assumed informally 5y several, members of a larger ^roup. We viewed these 
internal facilitatprs as "internal change agents," or the "^internal change 
agent team." The presence or absence of an "intarnal change agent" or 
"internal change agent team" appeared to be Bn important predictor of success. 

*See also Kell and^ouis (1980). 
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Tabto7-5 
TYPES pF LOCAL ACTION T€AMS 



ROLE 



FUNCTIONS 



FacWtotlno tMm 



Initiate meetings 

Plan agendas 

Spark the enthusiasm of 
participants 

Structure and facilitate the 
group process 

Collect and present informa- 
tion 

FoHow up on details 

Seek administrative support 
and cooperation 

Serve as the prirriary contact 
With external consultants 
and information resources 

Screen initial options 

Write first drafts of problem 
statements, search 
reouests, Implementation 
proposals, and plans 

Monitor implementation 



OPTIMAL SIZE 



REPRESENTATION 



TRAINING 




3 to 5 members 



Selection criteria 

• leadership ability 

• commitment to project 

• flexible time allocation 

• expectation of remaining 

m the system 
Candidates include the 
principal or assistant 
principal, central office 
representatives, and in< 
forma' opinion leaders on 
the staff 



Orientation to goals and 

process 
Special training in 

• leadership 

• problem solving 

• goup proce&s 

• formative evaluation 
« finding outside help 



DedtlofHiMUiia team 



Review information 
Brainstorm, discuss, and 

prioritize options 
Make final decisions 
Revise or approvMrafts of 
problem stateitMnts, search 
requests, imple^eritation 
proposals; aruj plans 
Review evaluation results 



8 to 15 members 



More teachers/implementers 

than administrators 
Representatives of every 

relevant grade level or depart^ 

ment 

Parents or community members 
only in some situations 



Orientation to goals and 
process 



Participate in training sessions 

about the solution 
Develop related materials and 

activities 
Participate in adapting the 

solution to fit the site's needs 

and context 
Implement the solution in own 

classroom 
Evaluate and provide feedback 
Recruit and train additional 

implementers 



Varies depending on the number 
of implementers 



Alt staff expected to implement 
in first stage, plus some who 
are expected to implement at * 
a later date 



Orientation to goals and 
process 

Special training in solution 
implementation 



A iieciBioo-m^inQ isam was one formed for tf ? purpose of reviewing 
infor«atior57 Fra ins terming, discussing and prioritising options (such as 
target problems, and solutions); making the final decisions (or deciding on 
recommendations to the administration); and revising or approving drafts of 
the^ problem statements, search requests, and implementation proposals ot 
plan«,. In some sites, the decision-making body was the faculty as a whole, 
while m other$ a smaller, though often representative, body made the 
decisions. ^ 

Finally, an implementing team was simply a trair^ed cadre or nucleus 
of implementors. The members participated m training sessions Bbout the 
adopted solution, developed related materials arid activities, participated 
in adapting the solution to fit the site's needs and context, implemented 
th^ solution m thear own claesrooms, evaluated or provided feedback on 
the solution, and occasionally helped m recruiting and training pdditional 
implementors. Although the implementing team may have had oxdcrable 
influence during planning for implementation, its members frenuently had 
no influfence at all durmq problem identification ^nd solution selection. 

Rased on our knowledge of the cases, as well as the quantitative 
findings presented above, it appears that the most consistently succei?sful 
model for team involvement was one m which a small, three- to f ive-mef^T^fcr 
group acted as a facilitating teaiti, while the faculty as a whole made vJie 
decisions. The second-best model for team involvement was one m #hich a 
decision-making team of .8 to 15 members adequately represented all grade 
levels and factions among che faculty, and established adequate procedures 
for communicatinQ 'With the faculty and obtaining faculty feedback (i,e,, 
through regular meetings between team members and constituents, news bul- 
letins, open team meetings, presentations at faculty meetings, and the 
^ like). 

So far, we have said very little about the role of school admanistra- 
torB in this process, beyond the fact that principal influence did not appear 
m the regression analyjos as a powe^^ul factor m tne explanation of site 
outcomes. In the simple correlations, principal influence is strongly 
associated with two measures of site success, organizational impacts and 
program incorporation, and associated to a lesser extent with process mcor- 
poratidn. Thus, the relatively modest explanatory power of principal mflu- 
f encfi, compared with some other aspects of the internal process, should not be 
taken as an indication that*the school administrator's role is unimportant. 
Even when decisions are made collectively, leadership is still important, and 
m many cases the principal is the most appropriAte person to provide that 
leadership. However, a distinction must be made between an administrator who 
facilitates group decision making and one who ^ dominates the decision-making 
process or even dictates the decisions. Boi rowing from both Thomas (1978) 
and Leith*^ood et al. (1978), we identified three types of school administra- 
tors: the facilitators; the directors; and the administrators. The facil i- 
tative leaders ware highly involved m the local problem-solving activities; 
they used a variety of strategies to involve teachers m decision making, 
and they relied heavily on teachers to influence other teachers. The direc- 
tive leaders decided themselves on the nature of needed changes and then 
tried to get their ' achers to follow their decisions. The administrative 
leaders were essentially passive observers" of the problem-solving process; 
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although they were aometiines authoritarian in matters dealing exclusively 
with adninistrat ion, they rarely became actively involved in the group 
decision-making activities. 

In our quelitative analysis of case materials, facilitative ^leader- 
ship was found to be the most consistently effective leadership style. There 
were also cases in w^ich centralized decision making worked vet-y Wwll. 
However, the directive leaders who were successful fell somewhere in the 
"grey area" between directive and facilitative leadership: they were attuned 
to faculty concerns, had good channels for communication with the faculty, 
and were trusted and accepted by the faculty as their leader in curriculum 
and instructional design. The passive administrators were generally ineffec- 
tive leaders of educational change. The degree of negative impact on the 
problem-solving process depended on whether the administrator was passively 
supportive of the process, passively opposed , or completely neutral . Thus, 
while the principal's active involvement in the process was not always 
essential, his or her support for the change program appeared to be crucial-- 
particularly at the point of implementation. Teachers seemed to feel that 
implementation was optional unless the principal backed the' change. More- 
over, In most schools the principals controlled school resources which would 
be needed for program incorporation. 

In short, the role of effective principals involved several responsi-- 
bilities which were not necessarily reflected in a strong degree of influence 
over decision mak^no. First, the most effective principals acted as internal 
change agents or facilitators , encouraging the staff to become involved in 
the group pro!jlem*"Solving process, assisting in the collection and presenta- 
tion of data for decision making, discovering additional resources (either 
through external change agents or on their own), leading or facilitating team 
meetings, negotiating special permissions from the central office or obtain- 
ing additional district funds, and responding quickly to any problems in 
implementation. Second, the principals in successful schools provided 
symbolic leadership . Even if they were not more actively involved, ^hey made 
it clear that they supported the group problem-solving process and, later, 
that they supported the innovation itself. This was very important to 
faculty who were used to looking to the principal for direction, as well as 
to faculty wno had grown mistrustful of the administration's commitment to 
teacher initiated change. Finally, the principals used their legitimat e 
authority in connection with budgets, schedules, and the superv ision of 
teachers to make sure that the necessary school resources were allocated to 
the selected programs and that these resources would also be available in the 
future.* 

The role of super intendern. s and other central office staff is 
more anbiguous in our analyses. On the one hand, increasing th^ breadth 
of involvement to include district administrators sometimes helped to develop 
a district-level commitment to the innovation, thus facilitating implementa- 
tion and incorporation of the new program. On the other hand, high levels of 
influence from upper-h I administrators occasionally detracted from the 
feelings of teachers that they themselves had played a major role in derision 
making, and thus reduced the teachers* own commitment to the innovation. The 



*In several cases , the principal • s legitimate authority was used to 
block, or reverse, changes; this is why it is so important to involve both 
teachers and administratoi's in the decision-makir^y process, or at least to 
make sure that the principal supports the decisions that are being made. 
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critical issue again appears to be the distinction between facilitative and 
directive leadershipi and between active and passive support. Facilitative 
leadership from diitrict administrators can be very effective m stimulating 
change at the school level; m addition, school-level changes often require 
continuing support from the central office if they are to survive. 

OTHER CHARACTERISTICS OF THE PROBLEM-SOLVING PROCESS 

As shown m Table 7-6, congruence with our criteria for "sound" 
group decision-making and "rational" problem-solving practices was found to 
be "great" or "very great" m over half the cases at each stage of the 
process. Moreover, as shown in Table 7-7, the rationality of the process is 
correlated with several of the site-level outcomes. Each stage of the 
process appears to be important, though the different stages are not always 
related to the same outcomes. 

Sites were most often scored low on problem-solviJig quality because 
of a lack of formalization, thoroughness, or objectivity m the problem-solv- 
ing procedures. This is true for both the "successful" and "unsuccessful" 
sites.* The difference between the "successful" and "unsuccessful" sites was 
m the reasonableness of the decisions themselves. For example, the success- 
ful and unsuccessful sites were equally likely to not consider alternative 
definitions of the problem* However, the unsuccessful sites more often 
failed to adequately specify the problem and more frequently developed 
a problem statement that was uncl^r, unmanageable, too narrow (trivial) 
or too broad (grbndiose). Both successful and unsuccessful sites tended 
not to evaluate solutions according to explicit criteria. However, unsuc- 
cessful sites were more often cited for lack of relevance or quality in the 
chosen product, lack of manageability- or cost-effectiveness, and for select- 
ing a solution before the forrnal search was completed. Finally, the failure 
to make formal plans or institute adequate controls over implementation 
occurred with equal frequency m successful 3nd unsuccessful sites. However, 
unsuccessful sites more often gave insufficient attention to planning for 
implementation and failed to gam or reinforce administrative support for the 
solution. 

In summary, thecfe^^c ^ of/ formal izat ion of the process was not 
as important as the appropriateness of the decisions. In many cases, critical 
school needs were obvious and consensus was reached fairly quickly on 
an appropriate — though perhaps not the best — course of action. In addition, 
the amount of thought given to planning for implementation could not always 
be judged by the existence of a documented plan. 

It is important, nonetheless, to be sufficiently deliberate, and to 
not move so quickly as to sacrifice the quality of the decisions. As shown 
m Table 7-7, the level of effort devoted to the process has a direct impact 



♦The categorical outcome measure described in Chapter 4 was used in 
classifying sites for this analysis. Thr^e m the "large-scale ROU success" 
or "mixed-high success" categories were logarded as "successful" sites, while 
those m the "moderate to low success" or "failure" categories were regarded 
as "unsuccessful" sites. A count was made of the times each criterion was 
noted as a reason for lowering a site's rating on problem-solving quality, 
and then the rank orders of the counts were compared between "successful" and 
"unsuccessful" sites. .-^ 
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Table 7-6 .^Z 

LEVEL OF EFFORT, GROUP DECISION-MAKING BEHAVIOR, 
AND PROBLEM-SOLVING BEHAVIOR BY STAOE 



Stage 



Problem 
Identification 



Solution 
Selection 



Planning for 
Implewentatign 



Implementation 



Level of effort 

Low (less than 10 person-days) 
Medium (10-30 person-days) 
High (over 30 person-Jays) 



305 

38 

33 



100% 
(nz88) 



2751 
29 



100% 

(n=C3) 



41% 
34 
23 
100% 
(n=76) 



34% 

29 
37 



100% 

(nr70) 



00 



-Congruence with group decisionr-fl>at<irK; 
practices 

To little or no extent 
To soiTie extent 
To a great extent 
To a very great extent 



Congruence with pyoblem-aolving 
practices 

To little or no extent 

To oome extent 

To a great extent 

To a very great extent 



19!S 
24 
34 
23 



100% 

(n=86) 



16% 
34 
35 
13 
100% 
(n;:B8) 



16% 

17 

34 

34 
100% 
(n=77) 



20% 
26 
38 
16 
100% 
(nr81) 



231S 
21 
28 
26 



100% 
(nz68) 



18% 
29 
32 
22 
100% " 
(n=73) 




100% 

(n=63) 



13% 
17 
36 
32 
100% 



(n=66) 
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Internal Process 
Vsnsbles by Stsge 



Itvsl of effort 



Problen Identif icst ion 
Solution Selection 
Planninq for Inpl. 
I«pleiiwntSt ion 
Total 



tjjnqruence with group 
McAionHiiilcinq practices 
Problem Identification 
Solution Selection 
Planninq for ImQi, 
I»plei»entat ion 
Total 



Congruence with problem-- 

•olvinq practices 

Problei! Identification 
Solution Selection 
Planning for Iiiv>l. 
Inplenentat ion 
Total 



Table 7-7 

PEARSON CORRELATION COEFFICIENTS 
FOR RELATIONSHIPS BETWEEN SIX SITE-LEVEl OUTCOMES 
AND THREE INTERNAL PROCESS VARIABLES, 
BY STAGE OF THE PROCESS 



Organizational 
Impacts 



• 16 
.15 

.21 

• 26* 



• 14 
.19 

• 19 

• 27» 

• 24» 



• 36»» 

.30« 

•40*» 



Product 
Incorporation 



^ite L^vel Outcomes 
Process 



Incorporation 



Problem 
Solved 



"Tcope""o7 
Implementation 



• 12 

• 10 

• 37*» 
.29* 

• 35»* 



,•05 
•01 
•01 
•06 
• 12 



-•01 

• 13 

• 17 

• 29* 

• 17 



.05 

• 11 
•08 

• 17 
•08 



.26» 
• 33»» 
.22 
.33»» 



.2> 

• 12 
.27» 

• 29* 

• 24» 



.06 
•05 
.18 
.10 
.14 



.07 
•05 
.14 
.20 
.09 



.18 
• 27* 
.27* 
.13 
.2/» 



• 13 
.22* 
.32*« 

• 28» 
.27» 



-•07 
-.08 
-•07 
•01 
•07 



• 11 
.14 

• 15 

• 15 

• 20 



•03 
-•05 
•09 
•00 
-•02 



•00 
•07 
.02 
• 10 
•07 



•04 
•05 
.06 
•00 
.10 



• P =< .05 



P < 



.OJ 
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on outconies; in addition, it is positively related to problem-solving quality 
(r = ,45, sig. .01). It is al^o related to breadth of involvement, since the 
more people who are involved the higher the level of effort for each activity 
(foi* example, r' = .48, sig. .01, for the relationship between level of effort 
and* breadth of involvement m planning for implementation). To achieve very 
broad and significant changes, it may be necessary to devote considerable 
effort to the process— as many as 30 or more persoa-days per year. In most 
eases, this would allow for weekly meetings of about one hour, or monthly 
meetings of several hours or more. 

While not over-simplifying the process, ^team momentum must be nfain- 
tamed. It is important not to allow the process to drag on too long, with 
long periods of inactivity between^ meetings. Teachers appeared to be most 
satisfied With the process when no more than one school year— or even less — 
was spent m analyzing problems and selecting a solution. Major steps toward 
implementation could then take place dufing the following year, while 
enthusiasm was still strong. 

SUMMARY 

^ The analysis presented above has produced a number of. findings 
that may have stxne significance for both theory and practice. Perhaps the 
most important overall finding is that, by themselves, variables describing 
the internal problem-solving process and the roles of various key actors 
m that process are of limited value m explaining RDU outcomes, as compared 
to product characteristics or external technical assistance. The one excep- 
tion to this generalization is equally important: process variables are of 
key importance m explaining the degree to which there is a change in organi- 
zational capacity, as measured by our overall organizational impacts outcome. 
In particular, the findings suggest, that broajj participation in both the 
adoption/selection of a new^ program, and m implementation decisions, is 
crucial tp achieving broad organizational development outcomes. 

A summary of some of the additional positive, practical findings 
that can be derived from the discussion m this chapter include: 

• Participating m a team-based problem-solving process 
can produce >signif leant staff development benefits for^ 

^ individual teachers. Encouraging meaningful partici- 

pation may be a sound strategy for school-based staff 
development, producing not only individual growth, but 
also improved classroom practice and materials. 

• Pa-rticipat ion in decision making by teachers does not 
necessarily reduce the influence of principals. In 
fact, principal and teacher influence tend to occur 
together. This suggests that principals should recog- 
nize that effective sharing of decision making with 
their staff members may augment rather than dimmish 
their ability to shape the curriculum and clasaroom 
practices in the school. 
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• Faculty participat iQn does not have to occur at all 
stages in the problem-solving process In order to 
increase school improvement outcomes. The most 
important stages at .which faculty involvement should 
be stimulated are inr adoption of new programs, and 
in the monitoring and assessment (^f implementation. 

t Having a well-organized decision-making team will not 
necessarily produce beneficial r^chool improvement out- 
comes. However, an active facil itating team which acts 
as an internal change agent for ^ school improvement 
effort, can be very effective as, loiig as there is also 
a high level of involvement by the overall faculty. 

• Whil^ attention to the quality of the problem-solviqg 
process was important (particularly in achieving broad 
organizational impacts), th^ actual activities engaged 
in were somewhat less important than the soundness of 
the decisions that were reached. Many non-formalized 
or ad hoc practices led to sound decisions, while elab- 
orate problem-solving activities sometimes covered up 
inappropriate allocations of resources and poor deci- 
sions. Thus, the emphasis on the quality of the 
proceTss and the movement toward school improvement 
objectives must be we 1 balanced. 

• Central officne and superintendent , influence on school- 
based problem solving tend to have negative impacts, 
and undermine important faculty participation. The 
reasons for this are not fuliy explicated in our data, 
but it seems that the perceived disjuncture between 
"top-down'* and **bottom-up" approaches to change focuses 
on the organizational boundaries between school and 
district, and not those between principal and teacher. 
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CHAPTER 8 
THE IMPACT OF "NON-PROGRAM" FEATURES 

INTRODUCTION 

The previous chapters have demonstrated that the intervention strate- 
gies employed by the R4D Utilization program were effective m producing both 
program goals and unintended school improvement (M^tcomes. Characteristics of . 
the externally developed products era the external human assistance and 
training were particularly important I'n achieving these outcomes. Features 
cf the internal problem-solving process were important as well. 

Although the RDU program involved a rather heavy level of effort on, 
the part of local school personnel , it was in large measure an external' 
intervention. There is an accunul ating literature, however, that suggests 
that local. site characteristics can be strong determinants of and/or impedi- 
ments to any school improvement effort, particularly those that employ 
external intervention strategies (Petersen, 1977; Derr, 1976; Weick, 1976). 
Some say that whether or not innovations get implemented is not a rational, 
predictable process, but is conditioned by critical events, "politics," and 
other features of the local context (Corwin, 1973; March and Olsen, 1974; 
Hage and Aiken, 1970). Organizational characteristics can overwhelm the 
externfiil intervention: the local culture, structure and characteristics of 
3taff and pupils are the major determinants of innovative behavior (Rosenblun 
and Louis, 1981; Herriott and Hodgkms, 1973; Zaltman et al., 1977). Thus, 
the most innovative schools and teachers are the natural "knowledge users," 
and the result is that good 'schools and classrooms continue to improve, while 
those that may be more m need of change fail to move toward new practices 
and ideas. Lack of experience and incentives for knowledge utilization m 
many school settings may result i'i increasing discrepancies between the best 
public schools and those that are Cup-rently unable to provide children with a 
high quality educational experience. 

In order to understand whether the RDU intervention strategies 
were really patent predictors of the school improvement outcomes that were 
evident m the participat^mg local school sites, it is iniportant to exploit 
an alternative explanation — the predictive power of local characteristics 
on those outcomes--and to compare the importance of local effects wjth 
those of the intervention strategies. The purpose of this chapter is to 
begin that exploration. The following questions are addressed: 

• What was the impact of local site characteristics 
on the school improvement outcomes?* 

• Who benefited from the RDU experience? In parti- 
cular, how successful was the RDU program m 
addressing issues of educational equity? 



♦The re^jtiv^ jjnpact of the intervention as compared to the site 
characteristics is discussed m the following chapter. 
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THE IMPORTANCE OF LOCAL CHARACTERISTICS 

C 

There are many organizational features and fortfes that can promote 
or inhibit a rational change process. Some of these are forces withm the 
organization's environment which it cannot control, such as community turbu- 
lenccy SCSy and demography. Others are features of the organization itself 
such as the characteristics of the student population and staff, the organi- 
zation's* size, and its structure and climate* Although all of these factors 
are relatively **non-manipulable,** some of the local features are more stbject 
to control than others. 

While many theorists have emphasized the importance of the effects 
of a s]^stem*s characteristics on change, primacy may be given to different 
categories of organizational characteristics m explaining change. (For an 
extensive discussion of these '^porbachesr see Rosenblun and Louis, 1981.) 
Some emphasize the importance of the school climate and the school staff 
(Likert, 1967; Katz and Kahn, 1966). Among these are some who emphasize the 
importance of cultural barriers to change (Sarason, 1971) and the relation- 
ship between "felt otJntrol" of participants and degree of mnovativeness 
(Zaltman, 1974). Others take a more structural approach suggesting that the 
complexity, authority struct^^e and formalization m an organization are 
predictors of change (Hage am Aiken, 1970). These are often viewed as 
constraints on innovative behavior (Blau and Schoenherr, 1971), suggesting 
that organizations function best when there is enough delegation of responsi- 
bility to accommodate the adaptation of broadly based organizational goals to 
local situations (Havelock, 1971; Thompson, 1967). 

Some forces precipitate change; others may be sources of resistance 
to change, although resistance to change may sometimes be healthy — such as 
when the proposed innovations may not meet the real needs of the local 
content (Zaltmaa and Duncan, 1977). Past mnovativeness is often viewed as 
an ufHi^ortant precursor to further change, due for example, to the cumula- 
tively increasing pressure on "non-adopters" to adopt (Rogers and Shoemaker, 
1961). This approach goes hand m hand with the view that the character- 
istics of the school staff are the important predictors of the change process 
(Corwm, 1973). The ways m which the environmental factors influence the 
change process have been variably a'^sessed. The environment has been viewed, 
on the one hand, as the direct force for change (Bowles and Gmtis,* 1972) and 
the changes are seen as a result of effective Response to changing environ- 
mental inputs (Baldridge, 1974). On the other hand, environment has been 
viewed as a constraint on change (Herriott and Hodgkms, 1973; Ottinger and 
Marks, 1974). 

While theorists do often assign primacy to part icul art organizational 
features, many, if not most, agree that thero are multiple characteristics 
that affect change iri education (e.g., Rosenblum and Louis, 1981; Zaltman et 
al., 1977). It IS important to consider the organizational context from the 
following viewpoints: the organizational environment, climate, structure and 
influence system; characteristics of individuals involved; and the nature of 
the intended changes. 
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VARIABLES AND MEASURES 



Data on the characteristics of the schools and pommunities involved 
m the RDU program Mere derived largely from questions on the surveys of 
principals and teachers. In addition, information from site visits and case 
Niateriaia provided by the projects were used to categorize the nature of the 
intended changes (i.e., the focus of the "problem" addressed by RDU), prior 
innovativeness, and the level of comft:'*niiy turbulence. These were coded by 
fliembers of the study team, using the consolidated coding folm described ° in 
Chapter 3. 

In keeping with the assumption that a variety of categories of site 
characteristics may be related to the innovative process, variables were 
measured m the following five categories: 

Characteristics of the community setting ; Var:^ables included an 
index of ' disadvantagement of pupils (based on percent of students one or 
more grades below grade level, percent qualifying for /ree or reduced cost 
lunch and percent minority), percent of students from white collar families, 
an index of recent changes m the community (including shifts in population 
and socio-economic changes) , and demographic nature of the surrounding 
community (large city, small city, suburban or rural). 

Principal characteristics ; The principal variables included the 
length of time the principal had been m the school, the number of years 
of teaching experience and prior administrative experience, membership 
ir> professional organizations, and the degree to which the staff rated 
him/her as an instructional leader. 

Characteristics of the teaching staff : \ These variables included 
information provided by principals on the following characteristics of the 
building staff — percent of teachers who were new to the school and percent 
who had be^ teaching ftit ten years or more, percent of teachers who were 
male, the pWcent with advanced degrees, and teacher reports on the number of 
professionarn5rganizations m which they were members. 

District and school size, structure and climate : Variables included 
number of schools m the district, number of pupils m the school, school 
complexity, school level, the influence of teachers, principals and the 
superintendent over key educational decisions,* staff orientation to change, 

^ 



♦Principals were asked to^rate the degree of influence of superinten- 
dent, principal, and teachers (using a 0-3 scale) on the 11 types of 
.educational decisions, including curriculum decisions, classroom activities, 
hiring decisions, identifying and implementing school-wide changes. See 
Appendix B. 
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collegiality, level of tension among staff,* and previous experience with 
similar problem-solving activities.** 

Nature of the problem being addressed by the RDU intervention : 
Variables included %i measure of the magnitude of the problem (centrality 
and severity) and whether the problem focus wis on pupil performance, 
the curriculum and materials, classroom organization, school organization, 
teaching effectiveness, pupil attitudes and behaviors and role relationships 
withm the school or between the school and central office or community.*** 

The incidence of "critical events" • that occurred m the schools 
and districts (such as strikes, ^he failure of a bond issue, or turnover m 
key positions during the RDU experience) were not systematically measured or 
coded m the consolidated coding form. However, since such events featured 
m several of the case studies and site reports, a separate analysis of 
the apparent impact of critical events on a subset of sites was condUcteu* 
The results of/fnis analysis are described later m this chapter. 

OVERVIEW OF THE CHARACTERISTICS OF THE RDU SCHOOLS 

Before discussing the degree to which local site characteristics were 
related to the school improvement outcomes, and before describing similarj- 
ties and differences between the "most successful" anc "least successful" 
schools, we present a brief overview of the kinds of -schools that partici- 
pated m the program. In general, they represented a broad range of school 
characteristics. About bb% of the schools were elementary, 205lj were middle 
or junior high schools,' and about 16S» were senior high schools. (The remain- 
der were special schools or not classifiable withm one of these three 
categories.) There were few "big city" schools (under 1%) . About lb% were 
m suburbs or medium-sized cities* Well over 60% wers m small cities or 
towns, or rural, with the large majority of these m rural areas. 

i 

The schools and districts varied m size. The average school was 
m a district with 13 schools, although the rang^ -/as from a district with 



*Teachers were asked to rate a variety of statements (on a four-point 
s^ale from definitely false to definitely true) reflecting orientation 
to change, collegiality , tensions, and the degree to which the pri>icipal 
was viewed as an instructional leader. Responses to items within each 
category were summed to creates scale r-epresentmg each of the varii)les. 
See Appendix B. 

**Two variables vere constructed based on a number of items rcrnrded 
on the consolidated coding form from site visit and site case materials. 
One scale represented the degree to which the site had favorable precedents 
for change. The second . reflected the degree to which the site had engaged 
m steps of the RDU problem-solving pro(?ess prior to the site's entry into 
the program. 

f , . 

***These ratings were based on the judgement of the core staff member 
coding the "problem characteristics" from site visit and case materials data 
and from site reports prepared for the RDU projects. 
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one school (with two teachers and 71* pupils) to one with 79 schools. The 
average number of pupils in a school was 388, although at least one school 
had close to 3000 pupils. Most schools had relatively enduring, stable, 
and mature staff. The average teacher had 19 years .of teaching experience, 
and the average school had over 40Si of teachers with ten years or more 
experience. About 7% of the teachers were new to their schools, although the 
range was from 0 to 38%. Many also had masters dejgrees (on average, -30% of 
the teachers in each , school ). About^ 28B of the teachers were male, most 
being in secondary schools. * 

Principals had been in their schools fran 1 to 27 years, with tjie 
average Being 7. Most had had prior teaching and/or administrative experi- 
ence. The average principal held membership in three professiowal organiza- 
tions. Only 2% reported no professional organizational affiliation. ^ 

The percent minority in the schools ranged from X) to 9Q% with the 
average being 145o. (Schools with high percentages of minority populations 
were largely in two of the sevafv^.tHfrTprojects*) The percent of students one 
-or more grades behind their grade level ranged from 0 to QQ% (the average 
being 24%). ^ 

Many schools had little or no precedent for being involved in an 
innovative program. Thirty-five percent had not been involved in federal or 
state-funded school improvement programs * other than Title I. More than 
half had no precedent for forming a local problem-solving team. About 20% of 
the schools seemed to have a favorable precedent with some aspect of. the RDU 
experience; very few ( unde , 5%) seemed to have experienced unfavorable 
precedents. Thus, overall, a review ^f the background and prior history of 
the RDU schools suggests that as a group, tbey were not necessarily "pre- 
destined" for success. 

The RDU schools became involved 1*1 the RDU program in a v«riety of 
ways. The recruitment process varied not only across projects but within 
projects as well.* In -»me cases a principal or inservice committee ac- 
tively volunteered, as ir. the cl e of a principal who asked to participate 
when she heard the p'^oposal was being submitted during a Teacher Education 
Center meeting (in the Florida project). In other cases, school personnel 
were looking for resources to support ongoing inservice programs or to deal 
with pressing problems and "stunbled" upon the RDU project in their search. 
Sometimes field agents recruited sites, either because they had previously 
worked with the sitl^ in other issues, or because they heard that ^^site 
might be receptive to assistance, having already conducted a needs Assess- 
ment . , 



♦During preliminary site visits conducted in the first year of the 
study (the second year of the program) attempts were made to ascertain 
jinformation cn how sites were selected or became involved in the program. In 
many cases, respondents did not know or disagreed on the actual recruitment 
or selection process and we were unable to construct a valid or reliable 
variable for this topic. This section is based on information we were able 
to gather during visits with sites or project headquarters. 
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In the largest nCmber of E\tes, it appears that the schools were 
••volunteered" by t^op-level district administration, largely because of poor 
performance on standardized tests. In' a few sites, ^however, the faculty were 
given the opportunity to vote on whether they were willing to participate. 

-There is ho clear evidence that the most successful sites were -those 
which volunteered or were most anxious to participate from the. beginning- 
What the evidence suggests is that schools don't have to be consciously and 
deliberately^looking for help in order to take advantage of it when it comes. 

, IMPACTS or .LOCAL SITE 'CHARACTERISTICS ON OUTCOME'S 

A f Trst step in the analysis of local site characteristics was^ 
iu examine separately the relationship of each category of site character- • 
iQtic variables to the school unprovenr. outcomes. The results of regres- 
sions of outcomes on each of these categories separately, had little or no 
expl anatory power. ' For both char'lact^istics of the community setting and 
prancipal characteristics , there wfere no regressions that explained as much 

as 15S of the variance in any dependent variable.* 

■» * ' 

Even bn the basis of simple corrjplations, few demographic character- * 
istics seemed to make any difference (see Table 8-1). In otheiNWorc(s, size 
or type of communitV (such as degree, of rurality, for example^, th^k degree to 
which there had been recent community change or^ turbulence, and i:he socio- 
efconomic status of the community did not di*scr iminate. The -tr^ly outcome 
which was significantly correlated with rurality was person^V impacts on 
staff (r= .19). 

"Readiness," in terms of favorable precedents for change, was also 
not an mportanjt factor. The only readiness variable that did correlate 
wa^ the one that measured the degree to which prior steps had been taken to 
identify and bec^in to deal with the prob"lem that was a focus of the^ RDU 
intervention. It appears that if the problem-solvir^g momentLsm had already 
begun, there was a greater chancy that the RDU intervention would achieve a 
variety of impacts. Thus "pre-RDU" activity was significantly correlated 
with, scope of implementation (r= .23), with both process incorporation (r= 
.24) and program incorporation <r= ,28) and with other organizational changes 
(r= .25). * ♦ 

The sTze of the "school did not matter either, although there was 
a tendency for schools In more complex districts to achieve more program 
incorporation (r= .15) and widespread organizational changes (r= .20). Nor 
was there any evidence- that schqol level was related to program impacts. 
Contrary to a popular assumption that elementary schools are more likely to 
benefit from a school improvement program than secondary schools, there were 
no significant correlations between school level and any of the outcome 
variables. 
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♦Because of the large nunber of theoretically and practically insig- 
nificant regression^, we have chosen to present only tables where the un- 
adjust^ Multiple R is greater than .15. 



148 

1G3 



T^ble 6-1 
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[he background and experience of the principals in the RDU schools 
had little or no significant influence on the outcomes of the effort. 
It didn't much matter if the principals were new to the school, or whether 
they had had much teaching or admiJiistrative experience. The only outcome 
that was significantly correlated with principal experience and tenure was 
personal impacts on staff. However, as is noted helow. the role of the 
principal in the structure and climate of the school was very important. For 
example, the degree to which the principal was perceived by his/her staff to 
be an instructional leader was significantly correlated with widespread 
organizational changes ( r= . 28) and with the degree to which the problem was 
perceived as solved (r= .15). 

In a regression of outcomes on characteristics of the teaching staff , 
only the- percentage of staff who were male contributed significantly to the 
explanation of overall organizational impacts. It is interesting to note 
that this relationship was a negative one, suggesting that male teachers, who 
were also more typically in secondary schools (r= .80), may be more "indepen- 
dent" and tesistant'to an external intervention and the kinds of collabora- 
tive efforts that we re a feature of the ROU pr ogr am . Simple cor r el at ions 
show that the percentage of staff who were male was significantly negatively 
correlated with all outcomes except prdcess incorporation (with which it had 
no significant relationship). Many of these men, particularly in the rural 
communities which were so prevalent ariuri^ the RDU sites, may have held second 
jobs, or had coaching responsibilities in the school, thereby also making 
them less available and possibly less willing to spend the time, necessarvy on 
RDU. 

More important to the explanation of school outcomes were the 
chai acter istics of the structure and climate of the schools and the nature 
of the problem that was being addressed by the RDU program at the local 
site. Four structure and climate variables did explain 155o of the variance 
in one outcome — overall organizational impact. Among thes§ are teacher 
change orientation, principal influence over decision making, and teacher 
influence over decision making (Table 8-2). Although the regressions of 
other outcomes on structure and climate variables had little or ,io signifi- 
cant explanatory power, many of these individual variables were significantly 
correlated with several outcomes. In particular, teacher orientation to 
change was positively and significantly correlated not only with organiza- 
tional changes (r: .30), but also with process incorporation (i= .19), 
personal impacts (r= .18) and with the degree to which the problem was 
perceived as solved'^d'r .20). Likewise, teacher influence in decision making 
was correlated not only with organizational impacts (r= .23\ but also with 
product incorporation (r= .21) and process incorporation (r= .29). 

The evidence suggests that the -c'ccess of a program .like RDU may 
be heavily influenced by teacher attitudes and by roles played by teachers 
in the ongoing relationships in a school. Much of the internal problem- 
solving activities describfjd m Chapter 7 involve the commitment and involve- 
ment of school staff. It appears quite likely that if such commitment and 
involvement are already in place in a school, the desired program outcomes, 
as well as spin-off effects, are more likely to be achieved. It is interest- 
ing to note that organizational impact is also significantly correlated with 
coUegiality among staff (r= .18) and with (low) level of tension (r= -.23). 
The degree to which member nip in teacher professional ^a^ixza Lions is 
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Table 8-2 

STANDARDIZED STEPWISE REGRESSION (BETA)^ COEFFICIENTS 
FOR THE RELATIONSHIP BETWEEN SCHOOL STRUCTURE 
AND CLIMATE VARIABLES, AND ORGANIZATIONAL IMPACTS^^ 

(N=116) 



Structure and Organizational 
Climate Variables Impacts 



Teacher Change Orientation • •34** 
Teacher Influence in "Decision Making ,15 
Principal Influence in Deci3ion Making ,20* 
Principal Viewed as Instructional Leader -.15 
Collegiality 
Level of Tension 



Multiple' .17 
Adjusted Multiple .15 



Beta Coefficients are presented only for tho^e variables 
which contributed to the reported multiple R . The 
selection process was stopped wheg additional variables 
failed to increase the multiple R by i% or more; 
the order of entry was unforced. 

Data are presented only for the outcome measure for which 
structure and climate variables explained at least 155o 
of the variance after the selection process was stopped. 

♦ p.> .05 

p.> ,01 
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related to organizational change, on the other^hand, is negative (p=- .15). 
These professional associations tended to be bargaining units, and in small, 
relatively rural schools (i.e., those which vyere most representative of the 
RDU schools), teachers who were members of the teachers' associations were 
anong the more "militant" and probably more resistant to the extra demands 
placed on them in connection with RDU activities. Principal professional 
membership (in organizations which were not likely to be bargaining units) 
was not significantly correlated with ^ny outcome maasures. 

The category of site" variables that had the most explanatory power 
in regression analyses was the problem focus, i.e., the characteristics 
of the problem that the sites dealt with in the RDU program. Three outcomes 
(organizational impacts, staff impacts, and the degree to which the problem 
was solved) had at least 15% of the variance explained by characteristics 
of the problem (Table 8-3). The most important of these were a focus on 
\ classroom organization, and/or a focus on pupil V)erformance. Not only were 
th^se variables predictors in the regression analyses, but they were also 
highly correlated with many of j outcomes. For example, focusing on 
problems of pupil performance was positively significantly correlated with 
, organizational impacts (r= .30), product incorporation (r= .26), scope of 
implementation fr= .38) and the degree to which the problem *vas perceived as 
solved (r= .36)'. Similarly, a focus-on classroom organization problems was 
Significantly correlated with all outcomes except process incorporation, and 
was especially correlated with organizational impact (r= .42). On the other 
hand, the focus 'on , problems involving "relationships'* (i.et, problems of 
relationships among staff, between staff and administration, between school 
and central office oY community) was negatively related to outcomes, and a 
focus on problems of pupii attitudes or behavior was unrelated to outcaheS. ^ 
The implication of this finding is that tfie RDU intervention waq r.iost useful 
in dealing vvith the more concr.ete, classroom-or lentedNand p^rhapVnfore * *i 
"tangible" problems. In part this may be due to the nature of the product 
tjase that was a feature of most of the RDU projects. These were largely 
curriculum products and/ or those dealing with reading or math management 
systems--in other words, products largely relevant to sites dealing with 
achievement or classroom managerent problems. If schools chose to deal with 
problems of a more affective nature, they were less iakely to find suitable 
solutions in the RDU projects' knowledge bases, and ^re less likely to 
achieve corollary spin-off outcomes as well. 

It IS interesting to note that deal ing with a pupil performance 
problem was negatively correlated with process incorporation ( r= 22) , and 
focus on classroom organization was unrelated to process incorporation. This 
suggests that attention to successfully solving a concrete classroom-oriented 
problem may also result in other impacts ( as, for example, organizational 
and staff development impacts), but may be least compatible in the short 
term with achieving greater capacity for RDli-like problem solving that would 
be indicated by process incorporation. 

In summary, site characteristics, when examined in separate cate- 
goriesi pioved to have weak explanatory power, with the exception of problem 
characteristics and t*^ a lesser degree the characteristics of the structure 
and climate of the school. 
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Table 8-3 



STANDARDIZED STEPWISE REGRESSION (BETA)^ COEFFICIENTS 
FOR ThC RELATIONSHIP BETWEEN CHARACTERISTICS OF THE PROBLEM 
AND THREE MEASURES OF SCHOOL OUTCOMES'^ 

School Outi^ome 



Problem 

Characteristics 



Organizational 
Impacts 



Problem 
Solved 



Personal Impacts 
(N=77) 



Magnitude of 
the Problem 



.08 



.11 



Problem m 

Pupil Performance 

Problem m 

Classroom 

Organization 

Problem m 
Relationships 

Problem ir. 

Curriculum Materials 



.2V 



.38** 



*.ll 



.33** 



.26* 



.24* 



-^.12 



.21* 



Problem m 
Pupil Attitudes 
and Behavior 



Multiple R^ 
Adjusted Multiple 



.28 
.24 



.22 
.19 



.18 
.15 



•f Beta Coefficients are presently only for those variables which contributed to 
the reported multiple R • The selectit)n process was stopped when additional 
. variables failed to increase the multiple R by 15 or more; the order of 
entr> was unforced. 

♦•f Data are presented only for the outcome measure for which problem characteristics 
explained at least 15S of the variance after the selection process was stopped. 

• P £ .05 
** P < .01 
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However, one further step was taken, v<^iich was to examine the com- 
bined effect on school ifl^pacts of the most potent site characteristic vari- 
ables from several categories (based on simple correlations as well as " the 
regression analyses). For this analysis the following variables were chosen: 
teacher orientation to change; teacher influence over decision making; the 
index of disadvantagement of students; school level; percent male staff; the 
degree to. which the problem-solving activities had begun prior to the RDU 
program (an index of "readiness"); and the identification of the problem as 
being one of ciassrcjom organisation or pupil performance. As Table 8-4 
shows, these variables do explain a relatively high percentage of variance on 
many of the outcomes, particularly product incorporation (R2= .45) and 
organizational impacts (R^^ .40). Personal impacts are explained the least 
by site characteristics (R = .16). 

Two variables stand out as entering into the equations for five of 
the six outcomes in these analyses: prior problem-solving activities and 
focusing on a problem of pupil performance. Problems in pupil performance 
were especially highly predictive of scope of implementation, product incor- 
poration and the deg-.ee to which the problem was solved. Once' again, how- 
ever, it was negatively related to process incorporation. Teacher influence 
in decision making, on the other hand, was highly predictive of both product 
and process incorporation, as was (although to a lesser degree) the indicator 
of readiness. The most powerful predictor of organizational impact was 
teacher orientation to change. The most heartening implication of these 
findings is that, unlike the demographic characteristics which were not 
discriminating variables, from the perspective of outcomes, the predictive 
Categories are the most "manipulable" of the site characteristics, and 
Jp^refore anenable to intervention strategies. 

One category of local site conditions which had variable effects 
and which has not been discussed yet in this chapter is "critical events." 
It IS to this issue which we now turn. 

THE:^ IMPACT OF CRITICAL EVENTS 

Events such as school strikes, abrupt changes in a school's financial 
condition, or turnover among key personnel happened frequently- enough in the 
sites we visited that we began to see them, as potent lally important -factors 
in the explanation of site outcomes, perhaps even more important than factors 
over which there is more control. To explore this u 2a mo 1 systematically, 
we reviewed the summary analyses of each site prepared by the senior resear- 
chers involved in the consolidated coding process. (These analyses included, 
among other information, the code'rs' perceptions of the key factors which 
either facilitated or impeded success at each site.)- We also reviewed the 
complete set of field notes from each of our follow-up site visits, conducted 
in the Winter and Spring of 1979. On the tlSsis of this review, tables were 
constructed which recorded the occurrence . of each type of event , the mflu-. 
ence of the event on the site's activities (whether positive, negative, 
or neutral/unknown), and the classification of the site according to the 
categorical measure of site success which was described in Chapter 4 (large- 
scale success, mixed high success, RDU success, spin-off success, moderate to 
low success, and failure). 
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Table 8-4 






STANDARDIZED STEPWISE REGRESSION (BETA)* COEFFICIENTS 
FOR THE RELATIONSHIP BETWEEN SCHOOL CHARACTERISTICS AND 
SIX MEASURES OF SCHOOL OUTCOME'S 




School 

Characteristics 






School Outcome 




Organizational Product 
Impacts Incorporation 


Process Problem 
Incorporation Solved 


Scope of Personal 
Imolementat inn Tmnurf a 


School Level 

Index of Disadvantagement 

Teacher Influence in 
Decision Making 




• 

.39** 


.34** 


.23 

.20 


Teacher Change Orientation 
ui S Male Teachers 


.52** 
-.28* 




.28* 




Prior Problem-Solving 
Activities 


.21 


.21* 


• 29* ' 46 


.23 


Problem in Pupil 
Performance 


.30* 


.47** 


-.31* A7** 


.40** 


Problem in Classroom 
Organization 






.27* 


.39** 


Multiple ^ 
Adjusted Multiple 


A2 
.40 


.50 
.45 


.31 .40 
.24 .34 


.40 .24 
.34 .16 


* ?hp%p?plfinn^"^^ presently only for those variables which contributed to the reported multiple 

The selection procep was stopped when additional variables failed to increase the Multiple R^ by IS or more: 
the order of entry was unforced. . k i/» ui muic, 


♦ p. < .05 

ERlCl p. < .01 
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The results confirm that critical events (events which have the 
potential for disrupting or altering local problem-solving activities or 
.their outcomes) are not uncommon. In fact, they happen so frequently that 
they may be viewed as "normal" occurrences m schools, even though they 
are^ generally unpredictable at the outset of a project. A critical event 
does not, however, invariably signal a crisis, or furnmg point, m project 
activities, as we explain further below. ^ 

Roughly hal f of the 90 sites examined encountered one or more of 
the following events during the two to three years covered by our data 
(the nunber of sites is given m parentheses following each event):* 

• Teacher/ implementor turnover (15) 

• Turnover m superintendent or key cfentral 
office staff (12) 

principal turnover (11) 

/ 

• /Change m financial conditions, usually for 

the worse (11) / 

• Field agent turnover (9) ^ 

• Team leader turnover (7) 

• Teachers' strike or other job action (5) 

• Reorganization of schools (5) 

• Change m district policies or priorities (4) 

• Reductions m force (4) 

• Crisis m parent/school relations (2) 

• Judicial intervention m the school system (2) 

• Severe storm (1) 

Over a third of the sites experienced turnover m local personnel — either 
a high rate of turnover among implementing staff or at least one changeover 
m principal, team leader, superintendent, or a key central office staff 
member. The t,urnover was almost always seen as having a negative effect on 
the local project activities; this was also true of most other kinds of 
critical events. 

Turnover among the implementing staff meant that new staff had to 
be oriented to the project and given training m use of the innovation. 



*Note that the real incidence of these events is probably even 
higher, since the data which were examined presumably mention only events 
that were perceived by our research staff to affect the project. 
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New staff often received less training, or less foimal training, than the 
original implementors. Many schools relied on over-the-shoulder or informal 
training by other staff members, rather than on bringing back the product 
developers or other outside consultants to conduct formal training sessions. 
New staff were therefore less likely to have an adequate understanding of the 
innovation. In addition, because* they had not participated m selecting the 
innovation or planning its use, new staff often were not as committed to the 
innovation as staff who had been involved in the decision making. If they 
were given any choice, new staff were less likely to use the innovation than 
the original implementors. The sites which coped successfully with turnover 
in the implementing staff were those m which the expectation for schoolwide 
adoption of the innovation was very clear, and in which training mechanisms, 
either formal or informal, were firmly established. 

Principal turnover %pemed to have its greatest negative impact 
on incorporation of t^e new problem-solving procedures — although some new 
principals were also less supportive of tr 3 curiicular innovation. Basics- 
ally, most principals had* their own leadership styles to which they were 
accustomed. In cases where thte new principals were already inclined to be 
facilitators, they welcomed the fact that staff had been trained in partici- 
patory problem-solving techniques. However, wtiere the new principals were 
inclined to be authoritarian or directive, the staff's experience in partici- 
patory decision making was unlikely to influence a change in their' decision- 
making style. "There was at least ' one' exception: at one site the staff 
insisted that the new principal continue the participatory process for future 
decisions. 

Curricular innovations were seldom killed outright by the new princi- 
pals, though they sometimes died for lack of strong principal support. New 
principals, like new teachers, were seldom deeply committed to the innova- 
tions of their predecessors. Again, there were exceptions: in two sites, 
the active enthusiasm of a new principal stimulated the use^of an innovation 
which ^d been given only lukewarm, or passive, support by the previous 
administrator. 

In cases where there was turnover in the "internal change agent" 
(who could be a teacher, a distract staff member, or a principal), the 
implementing staff tended to lose interest in the innovation. Even where the 
internal change agent was replaced by someone just as capable and just as 
committed to the innovation, there was usually a temporary setback due to 
the discontinuity in leadership. In certain situations, however, turnover in 
the formal team was beneficial: in one site, a reading specialist hired from 
outside the school district to coordinate program implementation was disliked 
by key staff members; the program was not really successful until after the 
coordinator moved on to another job and was replaced by a member of the 
original staff. 

A change in district financial conditions can affect incorporation 
of an innovation in several ways. For example, in several cases the dis- 
tricts were unable to come' up with the funds needed for replacement of 
consumable materials. In one district, where there were staff cutbacks, the 
remaining staff had heavier workloads and didn't feel that they could devote 
the extra time to the mnovatio' . In nearly all the districts where there 
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were budget cutbacks, the worsening financial conditions eroded staff morale 
and motivation so that it was more difficult to maintain staff enthusiasm for 
changes which required extra v/ork. 

While a teachers* strike was a serious setback m several sites 
(due to the time lost and negative impacts on teacher morale), m two sites 
the strikes led to even greater commitment to the innovation. The strikes 
reportedly resulted m greater cohesiveness and collegiality among the staff, 
as well as m a greater need among teachers to demonstrate their professionalism. 
Moreover, m one of these cases, the staff apparently regarded the hard work 
that went into learning to use the innovation as "occupational therapy" to 
help them forget the unpleasantness of the strike. 

School reorganization*, including changes m attend?^nce areas, consol- 
idation, OF changes m grade levels, had mostly indir^cl, although profound, 
impacts on project activities. For example, one elementary school dropped 
out of the RDU program when a shift in attendance areas caused a dramatic 
decline in the nimber of children performing below grade level. In another 
district, the site school was converted from a junior high school into an 
elementary school; the field agent tried to continue the project activities 
m the school to which most of the site's teachers had been transferred, but 
this was realJy like starting all over again, since she was workir^ with a 
new principal, a new reading coordinator, and a staff wt^iich waS, on the 
whole, quite different. In another district, attention was diverted from the 
project by the need to plan' for the inclusion of ninth grade classes at the 
high school. 

A shift m district policies or 'priorities can also affect the 
amount of time and attention- devoted to the project. For example, if a new 
basic skills currjculum is adopted soon after supplemental career education 
materials are purchased, teachers tend to ignore the supplemental materials 
until they have befcome comfortable with the new curriculum.. New district 
policies may also be m direct conflict with a recent innovation. For 
example, in one site the district adopted a very strict pupil progression 
plan which conflicted with the ungraded, continuous progress approach re- 
cently adopted m the site school. 

While this has been a fairly lengthy discussion of the consequences 
of critical events, it is important to note that critical events were rarely 
the deciding factor m the successful outcome of a site's involvement in the 
RDU program. We have just seen that an event which would normally be con- 
siderpd a ^e^^back, such as a strike or principal turnover, can be a positive 
factor under some circumstances. Moreover, even a critical event whose 
influence is clearly negative is just one factor among many; its influence 
may be swamped by countervailing positive factors, such as strong leadership 
or an innovation that is clearly effective. 

In the final analysis, we found no clear relationship between the 
occurrence of a critical event, or even a string of such events, and site 
success. Yet critical events were decisive m some instances, as the 
following cases illustrate. 
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No commitment . An elementary school adopted a career education 
package that consisted of materials and instructional aids to be worked into 
a teacher's daily instruction. Although the principal was the driving force 
behind the project, all derisions leading to the adoption were made by a site 
team consisting of six teachers, the principal, and a parent. After using 
the product for -a year, most faculty felt it was a good one which fit the 
school's nt.-eds very well. A formal evaluation also yielded results which 
were substantially in favor of the product. Yet two years later, use of the 
product was minimal — almost nonexistent except among team members. The 
dynamic principal who had supported the innovation had moved to another 
school; the new principal did not care if teachers used the product. The 
most committed teacher on the site team had also been transferred. Further- 
more, the district ran into financial problems resulting in larger classes; 
teachers said they barely had time to get through the basic text in each 
subject, much less to use the career education materials. Use of the 
product was optional, and not monitored. The product was seen as just 
another tool to be used or not, depending on time constraints and teacher 
inc I mat ion. 

The new principal was also used to being more directive. The tea- 
chers at this school had always been independent and had pushed for involve- 
ment in decision making. The school had a very emotional staff meeting where 
the staff pressured the principal for greater participation. According to 
his own reports, he has tried to change. On this he said: "I don't tell 
them where I want th^^m' to be, but slowly manipulate them to that goal, with 
their inputs along the way." 

The SI Iver I in inq. A junior high school adopted a reading improve- 
ment program consisting of a great variety of strategies, one of which was 
to irlvolve teachers of all subjects in'the teaching of reading. Identifica- 
tion of a problem and selection of the solution haQ gone smoothly, despite 
the staff's underlying distrust of the central office. However, the first 
year of implementation was disastrous. Firsts \he central office resisted 
the school's request for a full-time reading coordinator until late in 
August. The .hstrict reading specialist, who had guided the staff through 
the problem-solving process, resigned out of frustration in early June. 
Spurred on by the RDU field agent, tho new district reading specialist was 
able to obtain approval of the new position; but by that time the local staff 
had already been assigned, and a new arrival from out of state was hired as 
coordinator. The new coordinator was inexper lencad and, in addition, -became 
involved almost immediately m a turf battle with the head of the language 
arts department. She was also perceived as being on the wrorug side in a 
protracted contract dispute between the teachers' association and the central 
administration. She was never well accepted by the teachers and also felt 
she had no real support from the school principal. The project seemed doomed 
to failure after just one year. The turning point came when the reading 
coordinator left the school system and was replaced by the language arts 
chairperson. Although her approach was very low pressure, the language arts 
chairperson was able to get the project rolling. She also had the active 
support of a new principal, who was a firm believer in the importance of 
reading instruction. A prolonged teachers' strike at 'the beginning of the 
school year actually boosted teacher morale. They emerged from the strike as 
a cohesive unit (all but two had been involved) and devoted a great deal of 
their renewed energies to making the program work. 



In summary, although the consequences of critical events are van-- 
able, depending on unique circumstances, it is clear that they do place 
limits on the extent to vyhich one can rely on a rational, systematic, struc- 
tural approach to the management of change* Critical events which change 
site conditions happen almost routinely, and therefore the change approach 
must remain flexible enough to cope with, and adapt to, changing circtin-" 
stances* 

WHO BENEFITED FROM THE RDU EXPERIENCE? 

One objective of this chapter has been to answer the question, 
who benefited from the RDU experience? In particular, how successful was 
the RDU program in addressing issues of educational equity? Were schools "in 
need" able to benefit and acquire a greater capacity for knowledge utiliza- 
tion and consequent school improvement outcomes, or, as is frequently pur- 
ported, did only the good and more sophisticated schools get better, at the 
expense of those in greatest need? 

The previous analyses suggest that a number of local school charac- 
teristics were indeed significantly correlated with the intended and unin- ^ 
tended outcomes that were found to be associated with the RDU experience* 
However, the predictive variables tended not to be the most non-manipulable 
demograp^^ and descriptor variables such as those that would characterize 
the most advantaged, sophisticated, previously innovative, complex schools 
with many available resources* Instead, the predictor variables tended to be 
those which characterized structures, behavior and orientations of staff, and 
a focus on particular types of problems under the aegis of the RDU program* 

To further explore the question of who benefited from the RDU 
experience, an analysis was undertaken to examine the differences between 
the schools that were found to benefit most from the RDU program and those 
that benefited the least* For this analysis, we turned to the "categorical 
outcome" classification thai was described in Chapter 4* In that analysis, 
schools were grouped in six categories ranging from those which were low on 
all outcomes to those which were largely successful, or high on most out- 
comes, with the moderate, or largely spin-off successes in between* The two 
highest and two lowest groups of schools were compared to see if there were 
significant differences between the two clusters of groups on the site char- 
acteristics described in this chapter* The result of that analysis was 
largely confirmatory of the regression analyses. There were no significant 
differences between the two groups of schools on the demographic character- 
istics of rurality, size, index of pupil disadvantagement and SES, although 
there was a difference on percentage of staff who were male* (See Table 8-5.) 
For school problem characteristics, only the selec4.ion of classroom organi- 
zation as the problem focus — discriminated between the two groups of schools* 
Once again the category of variables on which the most successful schools and 
least surressful schools differed the most were behavioral and cultural 
features of the school, such as teacher influence in decision making, teacher 
orientation to change, the degree to which the principal was viewed as an 
instructional leader and the degree to which problem-solving activities had 
already begun This suggests that the capacity of local schools to engage 
successfully in problem^sol ving activities and achieve school improvement 
outcomes xs in large part a function of the power and influence relations, 
and cultural conditions within sites. 
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Table 8-5 



OirrERENCES ON SUE CHARACTERISTICS BETWEEN 
MORE SUCCES^rUL AND LESS SUCCESSFUL SCHOOIS 
^ IN THC ROU PR^pRAM 



Site VTiables ^ 

. • f ' 

Rurality (N=1095 

CommrMty Turbulence (N=67) 

Index of Disadvantagement v (N=98) 

S£S ^ (NrlOA) 

' Himbet of Schools iti District ^ (Ns58) 

Nuiber of Pupils^^m School ^ (NrllO) 

School level ' (N=lOl) 

Behavioral and Cul tural features ^ 

Teacher Inriuence m Decision Making (N=97) 

Principal In^'luence m Decision Making (N^96) 

Superintendent Influence m Decision Making (N=96) 

Teacher Orientation tj Change ^ (N=l26} 

Collegiality (NnZS) 

f^cedents for Change J "read iness'M (N=U7) 

Prior Problem-Solving Activities {N=7l) 
(pre-RQU "readiness") ^ 

Principal as Instructional leader ' (Nr'126) 

* Male Staff (NrllO) 
School Problem Characer ist ics 

Severity of Problem (N=147) 

Pupil Performance Problem (7^=71) 

Classroom Organization Problem (Nr7l) 

Staffing Problem t (N=7l) 



r Value 



.86 

.28 

.00 

.60 

1.32 

1.12 

/ 
2.38 

7.9S 

.40 
7.40 
.47 
.10 
5.12 

8.23 
7.66 

.06 
.52 
4.43 
1.61 



Significance 
level 



.36 
.60 
.99 
.44 
.25 
.30 
.13 . 
» 

.006^ 

.18 

.53 

.007»* 
.49 
.76 
.03* 

.005»» 

.007**, 

.80 
.47 



.04* 
.21 



• p. < .05 

p. < .01 
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Overall, the disadvantaged, more rural, less innovative schools 
we e just as likely to achieve knowledge utilization outcomes* m the RDU 
program as their more "advantaged'* count erparts* host local site charac- 
teristics did not make a difference, nor were they a handicap. Thus xf one 
views "equity" as the ability to achieve program outcomes regardless of 
external and non-man ipulab le conditions, a dissemination strategy such as 
that pmployefi bv the RDU program can address issues of educational pquity. 

SUMMARY 

At the beginnLoq of this cha^jter we indicated that theorists of the 
impact local school characteristics on "^nnovat i veness" or successful 

compIe?tion of a school improvement program could be clas ified into a number 
of groups--those who stress the climate or culture of the school, thoStT 
who emphasize structural features, those who believe that staff characteris- 
tKS will affect behaviors most significantly, those who believe in the 
importance of contextual characteristics (including student SES "and achieve- 
ment ipveis) and thore who examine the characteristics of the intended area 
of char^ge. Our data suggest that, at least among fhe schools studied in this 
program, two of these groups have greater cplanatory power. Variables 
f'^easurinc tne schools' ci im^te for innovation were quite significant in 
(Explaining jhanqe » with particular importarce being accorded to variables 
t.^ppmg the teachers' orientation toward change^ and the experience of the 
^scnooi in prior problem-solving activities related to the problem in gues- 
: inn. Also important m explaining sctiool outcomes was the natur e of the 
pcob i em : a focus on classroom organ z?3t ion and/or pupil achieverr?nt tended to 
^e associated with higher levels of chant^e. However, other appro£:ches also 
riad sor^ merjtt. For example, structural feat ures of the school, particularly 
the degree to ' h teachers influenced the"decision-makinq activities, 
affected thv uutL^^es of the change program. 

Unlike many other studies, our data do not indicate that the demo- 
Cj^r aphic cnpract er j st ics of schools, inciudirK| principal demographic charac- 
teristics, teacher demc^graphic chdracterist ics, or student characteristics 
have profound impacts upon the outcomes of participating in an innovative 
proqr irri. We found this to be a finding of some practical significance, since 
in most studies of "naturally occurring" innovation these non-man ipulab le 
f^jrtors havf^ major impacts. 

These findings suggest that local site characteristics that affect 
the outcomes of the change process are those that may, m fact, be mc- A 
susceptible to change themselves. T^-ijs, it is difficult to alter the level 
of a schoo i - -whe t he r it is serr^n-jary or elementary. However, teachers 
with attitudes that are unsupportive or change may, in fact, be made more 
positive if they are given reason to believe that their efforts will 'be 
rewardf^d and will produce sofiethmg of value. Similarly, since schools 
typical iy have an endless supply of problems that could be "managed," it is 
rei^itively simple to begin a m*: jor change pi ogram by emphasizing issues that 
ff^iate to {^assroom and pupils,- 

Uther f inding:^, however, indicate a set of school contextual 
^eature^. that mav be difficult to arit icipate dv manage, yet have significant 
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consequences for planned change activities. We refer here to the " normal 
cr it ical events " wh ich occur each school year , and wh ich both d isrupt and 
enhance the educational performance of schools. While these disruptions tend 
to be perceived negatively by those who experience them, in our data many 
unpleasant critical events had ultimately positive (or at least ' nsut ral ) 
impacts on the school improvement activities fostered by RDU. In an equal 
number of cases, particularly where there was turnover amc key staff 
members, serious disruptions could occur. As increased cutoacks in funding 
for education threaten the stability of school staffs, and also increase the 
rate of critical events in schools, the prospects for planned change may be 
dim inished. 

An 'important question remains. While local site conditions were 
predictors of school improvement outcomes, was their potency greater 
[ens than the pov-'pr of the intervention'^ [his question is addressed in the 
chapter which folK'^s. 
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CHAPTER 9 



CONTLXf AND INTERVENTION: THE POTENCY OF MANAGED CHANGE 

INTRODUCTION 

In the previous chapters, we have seen that each of the three 
elements of the RDU intervention--the use of externally developed and 
vai ated products, external technical assistance, and ar: internal problem- 
solving process that promoted rational decision making and broad participa- 
tion of various constituencies--had a measurable i.npact upon a variety of 
school improvement outcomes. We have also seen that the characteristics 
of the school and the problem context were strongly associated with school 
outcomes as well. Turning again to the schema (originally discussed in 
Chapter 2) which has guided this research (Figure 9-1), it is evident that 
the analyses presented in the previous chapters have only deali with a 
limited number of its presumed rel ationships ( i .e . , those represented by 
arrows niinbered 5, 6, 7 and 8). The previous analyses have not dealt with 
indirect effects on the school \mprovement outcomes^ such as the hypothesized 
relationships regarding the ways in which the strategies of the intervention 
ran be reinforced, either by design (arrow 4), or within a dynamic process 
(arrows 2 and 3). Nor have we examined as yet the ways in which local 
conditions can affect the internal process (arrow 1). in order to examine 
the more dynamic aspects of the general model, we must move beyond the 
ultimate RDU school improvement outcomes, to look at the ways jn which the 
mtervent ion processes interact with or feed in*^o one another. 

In addition, the previous analyses have not answered two more global 
questions that are not portrayed ^n the s6hema: Does the ROD approach to 
knowledge utilization and school improvement consist of three separate 
strategies, or does the combination of strategies have potency over and 
above the separate components'^ And, finally, does the support of a managed 
cnange approach through a combination of intervention strategies have any 
impact on school improvement over and above that accounted for by features 
of the school'' 

Thus, the purpose of this chapter is to "round out" the analyses 
that are depicted in the model, and to direct attention to competing hypo- 
theses or alternative questions about how school improvement results are 
In particuigr, the following questions are addressed: 

To what c^qree are t bre^ 'int ernal proolem-solv ing processes 
affected by the general c. haract en stdcs of the school 
(arrow i), and the type of external assistance (arrow 2)^ 

lo what degree do internal probld|m-sol vir.g processes 
at the site account for the qualify, validation, and 
difficulty of implementation of the products that are 
chosen (arrow 3)'^ 

Do project design features have an effect upon the types 
of services; provided to schools, particularly those that 
are most predictive of site success (arrow 4)*^ 
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FIGURE 9-1 

A SCHEMA FOR EXAMINING THE IMPACT OF KNOWLEDGE UTILIZATION ON LKAL SCHOOLS* 



LOCAL CONDITIONS 




i 



•Lines and arrows represent hypothesized relationships which are reported m this volune; 
dotted ii ,cs and arrows represent hypothesized relationships Which are discussed m other 
O reports of the RDU study (Corwm, 1980; Louis and Rosenblum, 1981). 
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• To wh*it degree do the three RDU strategies for change 
in combirvation have effects over and above the po^^er of 
each separately (arrows 5, 6 and 7 together)? 

• To what degree do features of the RDU intervention have 
an impact on schools over and above that explained b^. 

locgl site characteristics (arrows 5, 6 and 7 versus ^ 



Our model for exaroininy the RDU program assumed that the internal 
problem-solving process was a function of both the characteristics of Ihe 
local school and its setting (arrow 1) — particularly such features of the 
school as teacher change orientation, the degree to which teachers influence 
decision making in the school, and the "readiness" of the school as evidenced 
by previous problem-solving activities — and the training and support for 
problem solving that were provided by the project (arrow 2). In order to 
examine the impacts of these groups of variables on the problem-solving 
behaviors of schools, we first selected four indicators of the internal 
problem-solving process , based on their utility in other an?3lyses and their 
correspondLnce to the expressed objectives of the RDU projects: 

§ extent of faculty influence in problem solving; 

c breadth of participation .in solution selection; 

• breadth of participation in implementation; and 

• overall quality of the problem-solving process. 

These variables were then regressed upon eight site char;.cteristics — school 
level, percentage of disadvantaged pupils, teacher influence over decision 
making, teacher change orientation, school size, prior activities in problem 
solving, and w^<ether the problem focused on pupil performance or classroom 
organization — and eight external technical assistance variables — field agent 
initiative, field agent time spent with .the team, field agent contact with 
the principal, field agent mnovativeness, political and structural perspec- 
tives of the agent, amount of specialized training, and variety of trainers. 
The results of this stepwise regression are presented in Table 9-1. 

Several findings are revealed by this table. First, in combination, 
site characteristics and external ergent characteristics are moderately 
good predictors of the internal problem-solving behaviors of schools. More 
than 25% of the variance in each dependent variable is explained, using the 
adjusted multiple R statistic. Th^ level of explanation is highest m the 
case of the quality of ilecisior making in the problem-solving process, where 
the adjusted multiple R is .34.* 

2 

♦These adjusted multiple R s are considerably higher than those 
achieved when the regression is performed separately for site and external 
assistance variables, which indicates that the two groups of variables in 
concert are of importance m determining the outcomes of problem-solving 
activities (analysis not tabled). 
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PREDICTING THE INTERNAL PROBLEM-SOLVING PROCESS: 
SCHOOL AND PROGRAM EFFECTS 
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Tttble 9-1 

STANDARDIZED REGRESSION COEmCIENTS* FOR 
RELATIONSHIP BETWEEN INTERNAL PROCESSES 
AND SCHOOL CHARACTERISTICS AND EXTERNAL ASSISTANCE 
(N = 45) 



Internal Proccrss 



SCHOOL CHARACTERISTICS 



t'aculty 
Influence 



Sreedth of 
Involvement in 
Solution Selection 



Breadth of 
Involvement m 
Impl e mentation 



Quality of 
Problem 

Solving 



School Level eleVaecond) 
% DiMK^ventaged 
Teacher Influence 
Teacher Charge Orientation 

School Si2e 

Prior Problem* Solving Activities 
Pupil Performance Problem 
Classroom Organization Problem 



-.27 



421 

4 

.29* 



.17 



.19 
-.17 

.31« 

-.19 



-•41« 



-•30 

-.24^7 
-•21 



EXTERNAL ASSISTANCE VARIABLES 
Agent Initiative 
Agent Tiac on Site 
FA Political Orientation 
FA Structural Orientation 
FA Innovativeness 
Agent-Principai Contact 
Amount of Training 
Verict/ of Training 



V 



.72»* 



.23 
.23 



.13 
.22 

.14 



.18 



.64** 

.29 
,34* 



Multiple K- 
Adjus<-cd Multiple 



.33 
.26 



.40 
.27 



.42 
.29 



.46 
.34 
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•P i .05 
•♦p < .01 



^ Beta Coefficients are presented ordy for tnose variables which contributed to the reported 
multiple R . The selection proceas was stopped when additional variables failed to 
increase the Multiple R by 1* or more; the order of entry was unforced. 
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Overall, however, the patterns of relationships for most individual 
variables are relatively weak and/gj.- scat tered , while in several cases 
variables appear to have opposite irffpacts on different aspects of the problem- 
solving process. 

Thus, for example, secondary schools have lower levels of faculty 
influence, but broader involvement in the process of solution selection. 
This suggests that central office personnel ..ay become more heavily involved 
in determining the nature of the product to be implemented--but at the 
expense of broad participation within the school. Of course, in the case 
of secondary schools, many of the products we^ designed to be implemented 
by a small subset of teachers- -a specific deparWnent (basic reading programs 
were often limited to SQCial studies and language arts departments) or even 
to a few individuals (career education programs which focused on guidance or 
career experience activities). In these cas6s, the decisions often involved 
major staff allocation and scheduling questions tt^iat were quite different 
from thp aHnpt-jnn of a schnnl-widp reaHi.nn mananenent program in an ele- 
mentary school. In sumntary, elementary and secondary schools do appear to 
behave quite differently. 

Another example of inconsistent relationships between a school 
characteristic and the internal process is the positive association between 
the school's prior attempts to solve the problem and the breadth of involve- 
ment in solution selection and implementation, but the neqative (although 
non-signi f leant ) relationship of prior activity to the quality of the 
problem-solving process. Schools that had already er.qaged in significant 
activities to solve their identified problem had typically turned to central 
office personnel for as3ist3nce. Thus, these individuals were already 
involved m problem-solving activities — and had often already determined the 
nature of the problem and perhaps even the solution they wished to implement. 
In a number of cascS in our intensive sample, the RDU process was simply 
superimposed upon the pre-exis*: ing decisions sometimes by the central office 
and the principal and sometimes by a broader coalition of actors. In these 
cases the RDU program was most often used "opportunistically," to help carry 
out a decision that was already made, and one which was often not oased on 
the rational decision making model promoted in the RDU program. 

The role of external technical assistance appears to be less formid- 
able in predicting the breadth of involvement in solution selection and 
implementation than ocal site characteristics. However, it is of approxi- 
mately equal weight in its impact on the quality of the problem-solving 
process. Again, as in Chapter 6, we find that field agents who spend 
as much time as possible in actual interaction with the local problem-solving 
team, and those who adopt relatively conventional approaches to the innova- 
tion process have much greater impact. Training variables, which were 
extremely potent predictors of actual school outcomes, are of less importance 
in predicting the internal process: we find that the amount of training is 
related to overall faculty influence in the problem-solving process but, as 
noted in Chapter 6, this finding is somewhat confounded by evidence from the 
qualitative site data, which suggest that faculty with more influence tended 
to look for and demand more training. 



A final observation of mt rest that there is a clear pattern 
suggesting that school factors which do not predict the ultimate degree 
of "success" may still have a strong impact on the quality of the problem- 
solving process. Big schools and schools with a high proportion of disad- 
vantaged children exhibited relatively poor dec .sion-making behaviors in the 
context of the R&D Utilization program. Yet, as noted in Chapter 8, they are 
no less likely to a- hieve othei^ benefits. 

Overall, these findings suggest several conclusions First, although 
we ha^e learned that the nature of the problem-solving process is important 
to determine school improvement and knowledge utilization outcomes, our 
ability to ouxiel the factors that affect the quality of the problem-solving 
process, and the degree to which it involves participation both of faculty 
and other t^ey decision makers, is far from satisfactory. Second, the data 
suggest that, in pursuing additional explanations, levels of participation 
and influence are probably best explained by characteristics of the' local 
setting and the type of problem that is being addressed, while the general 
rationality of the decision-making process is somewhat more amenable to 
external intervention. 

A second set of analyses was conducted in order to examine the 
relationship between external assistance and the internal process (arrow 2). 
We had initially hypothesized that there would be an interaction effect 
between the role o^" the field agent and the role of the "internal change 
agents," mainly the principal, but also possibly central office specialists. 
It appeared, based on our field visits, that in many cases the field agent 
^role was a compensatory one. When there was a very active "internal change 
'agent," the field agent's role was more minimal and less initiating, but 
when there was no one playing an internal leadership role in the change 
process, a field agent often filled in. We conducted an extensive "search" 
for int^*ractiori effects, using variables measuring principal influence, 
central office specialist influence, and faculty influence, in interaction 
with field agent initiative and field agent time. However, not a single 
two-variable interaction proved to contribute significantly to the explana- 
tion of any school improvement outcome.* Thus, we are led to the conclusion 
that the relationship of internal to external change agents is, on the whole, 
an additive one. 



^x'^The search for interaction effects was conducted as follows. First, 
a pai]^of terms that were expected to interact was selected. An interaction 
term hbs calculated by subtracting the mean from each variable, and multi- 
plying them. The main terms were entered in a regression first, with the 
iriteraction term entering in a second step. Interaction terms would be 
considered siqruficant if they (I) had a regression coefficient whose F 
statistic was significant at the .03 level; and (2) contributed at least 
2 percent to the multiple P^. of the dependent variable. if interaction 
terms had turned out to be significant, then further explication through 
cross-tabular analysis would have followed, as done in Rosenblum and Louis, 
' 1981... 
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PREDICTING PRODUCT QUALITY: THE IMPACT OF 
INTERNAL PROCESS ON SOLUTION SELECTION 



The model presented m Figure 9-1 hypothesizes that the internal 
probletn-solving process had considerable impact upon the qualitry of the 
products that were selected for implementation (see arrow 3). In addi- 
tion, we have argued that the negative relationship that was found in Chapter 
5 bet*#een adaptation of the products and school improvement outcomes was a 
consequence of the attention paid to matching the characteristics of the 
proposed new practices to actual needs — a process that is not always evident 
m organizational decision making. In this analysis, we seek to examine the 
impacts of the problem-solving behaviors in the school (level of effort and 
problem-solving quality), the roles played by various actors (principal 
influence, faculty influence, superintendent influence, other central office 
influence) and the breadth of involvement in decision making (solution 
selection and implementation) on product characteristics. The following 
product characteristics were selected because they effectively predict at 
least some school-level outceme*^, zrJ Lcc?»»<5e they clearlv reflect some 
dimension of product value: 



• the quality of the product (an index made up of the 
teacher's perception of the novelty the ideas pre-- 
sented, their applicability in the clas9«;Qom setting, 
and the adequacy of information about how it could be 
implemented ^ ; 

• the degree to which the product required major changes 
for the classroon teachers 'difficulty of implementation); 
and 

• whether there was evidence that the product had been 
field tested or validated. 



A final product variable was selected to reflect the process of implementa- 
tion: 



The results of a stepwise regression including these variables is presented 
m Table 9-2. 

One statement that^ can be made quite clearly based on the relatively 
low adjusted multiple R s is that the data do not support the program 
design premise that a good internal problem-solving process results in the 
adoption and implementation of the products with„.^S^cteristics that are 
predictive of the school improvement outcomes (such ^ as product quality, 
difficulty of implementation a.id ^field test/validation status). In each 
regression, even the unadjusted Rs are quite small, particularly compared 
with those that have been found m other analyses. Only in the case of 
post-implementation adaptation do we find a level of prediction which meets 
the rough standard of policy and theoretical significance that we have used 
m prior analvses (that the adjusted multiple R should be at least .13). 



• post-implementation adaptation, or the amount of cha^'t^g" 
made in the recommended product or practices after imple- 
mentat ion oegan. 
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Tabl« 9-2 



STANDARDIZED REGRESSION COEFFICIENTS* FOR THE , 
RELATIONSHIP BETWEEN PRODUCT CHARACTERISTICS AND INTERNAL PRXESS 

(Nsl}7) 



Internal 

ProcMS 

VftTiable 



Product 
Qimllty 



Product Characteristics 



Difficulty of 
liiplsiseotstlon 



Field Test/ 
Validation 



Post-IiKpleiientation 
Adaptation 



Level cf Effort 

Quality of Problem- 
Solving Proceaa 

Faculty Influence 
on ProceTO 



.19 



.11 



.13 



Principal Influence 
on Process 



.24* 



Superintendent Influence 
on Process 

Other Central Staff 
Influence an Process 

Breadth of Involvement 
in Solution Selection 

Breadth of Involvement 
in Inpleflentation 



-.23 
.29* 
.15 



• 20 
.33* 

.17 



.19 



.20 



.16 



J6 



.24 



Multiple R^ 
Adjuateo Multiple 



.15 

.10 



.16 
.10 



.17 
.11 



.28 
.19 



•p S .05 

< .01 

♦Bets Coefficients aj^e presently only for those variables which contributed to the 
reported multiple R . The selection process was stopped when additionsl variab'*r 
failed to increase the Multiple R by 1% or more; the order of entry wss unforced. 
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However, despite the unimpressive power of the equations in predict- 
ing product characteristics, some of the individual patterns of relationships 
are useful in illuminating the impact of process on product selection. Two 
patterns which stand out, and help to explain the role of administrators and 
specialists in the process, are those involving the influence of central 
office specialists and the principal . Earlier, in Chapter 7,, we noted that 
the role of the principal in determining school outcomes was surprisingly 
limited given the significance attributed to that role in the literature. 
We also posited, however, that the principal's role was less direct in the 
RDU process because the more effective principals encouraged high levels of 
faculty influence and ownership. Here we see a confirmation that the pi-nci- 
pal*s role may be a more supportive and indirect one. Principal influence 
has particularly strong and significant impacts upon the selection of field- 
tested or validated products, and is negatively related to adaptation 
duririg the pos t- imp lement at ion period. Ihis suggests that the role of the 
principal may be important in internal quality control , both in the initial 
selection of a solution, and in maintaining a school-wide focus--i.e., 
providing an environment which discourages a great deal of localized class- 
room adaptation. > (Most post-implementa'.ion adaptation, according to data 
gathered in site visits, involved individual teacher adaptations rather than 
actual redesigns of a school-wide program.) 

The role of the cerU.ral office specialist also emerged as relatively 
important in many of our s}te visits, yet had little direct impact in pre- 
dicting the school improvement outcomes. Again, however, these data suggest 
that the specialist's role may be strongly weighted toward influencing the 
qualitv and significance of the solution selected, providing support for 
faculty T.embers to attempt more than they might have otherwise, and encourag- 
ing the selection of pro(^ucts that are well matched to the needs of the 
school. The f^t that the central office specialist's role also emerges as a 
factor in pr^^ictmq the f leld-tested/va I idat ion status of the product 
reinforces this view. 

Two other firxlinqs are of interest, largely because they confirm 
findings that emerged from the site visits. First, as we expected, the 
quality of the problem-solving process is affirmed as a mechanism for reduc- 
ing the- requirement for post-implement at ion adaptation. Wnere the process 
has been attentive to the more rational aspects of problem-soUing — including 
techniques sucti as developing specific criLeria against which to assess 
diternative solutions to the problem, conduct mo serious assessments of 
alternatives, developing lelatively detailed plans for implementation 
■including plans for training, monitoring implementation, and correcting 
any pioblems that ar ise) --extens iv e adaptations are typically not necessriry. 
This finding does not, of course, imply that the implementation process was 
not d dynaoiic one. In fact, se K-cor rect ing feedback was pai t of the defini- 
tion of a high quality problem-so Iv inq process against which the school's 
activities were measured. However, fine-tuning should not be confused with 
rhanges which altei basic features of the intended innovation, nor should ad 
hoc adaptation by individuals (which may, cumulatively, have the effect of 
changing the intended innovatiori) be confused with planned adjustments. When 
adjustments were made m many of the schools that ranked among t,he highest on 
the quality of their problem-solving processes, they were accompanied by 
consultation oi further training from the developer oi othef consultants. 
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Another interesting finding within Table 9-2 concerns the role 
of broad involNrement in implementation. Implementation involvement makes a 
positive contribution to overall school improvement outcomes and is also 
positively related to post- imp lamentation adaptation (p = .055). We believe 
that this relationship, although weak, highlights a problem of transition 
which occurred for many of the schools, and which was discussed in detail in 
an earlier report (Keii and Louis, 1980). This dilemma concerns the change 
from a relatively small facilitating team to a more broadly based decision- 
making or implementing team. Because this transition often involved sub- 
stantial increases in the ntinber of '^actors'* who were expected to partici- 
pate, the additions of new agendas and issues raised problems for the 
"quality control" of the process. We believe that this problem with transi- 
tions accounts for the apparently positive relationship between broadly-based 
influence over implementation decisions and post-implementatiOn adaptation. 

PROJECT DESIGN AND MANAGEMENT AND THE DELIVERY 
OF SERVICES IN THE RDU INTERVENTION 

Prior sections of this chapter discussed the degree to which the 
intemal problem-solving processes were affected by the characteristics 
of the school (arrow 1) and the type of external assist nee that they 
received (arrow 2). We also described the impact of the internal decision- 
making processes on the characteristics of the products that wei'p chosen 
farrow 3). In this section, move to the next question implied by the 
scheme depicted in Figure 9-1: Do project design features have an effect 
upon the types of services provided to schools, particularly those that were 
most predictive of site success (arrow 4)'^ 

The RDU program, like many other federal and state-finded school 
improvement programs (particularly those that provide indirect support) 
assumed that the management of change occurs not only at the local level, but 
at a broader project l^vel. It is through projects, state departments, or 
other service agencies providing suppoit that the services to be delivered 
are designed ana coordinated. The RDU program announcement (NIE, 197^) 
explicitly assurred that the projects rould directly affect both the products 
and other infoimat ion that would be made available to schools, and also the 
various types of technical assistance, training or other human resources that 
would be provided. 

Uthei recent theories of change and implementation, however, might 
cause us to question whett^ei program design features will be translated 
into the types of services expected at the local level. Many studies, 
including oui own, have pointed out how there is considerable "slippage** 
betwee e design of a program and what actually occurs at the service 
deiivei; level (Ym, Gwaltney and Louis, 1980; Cotwin, 1981). In. addition, 
analyses of project structures such as those in the RDU proqrari have used the 
'imagery of the ''dispersed organization*' (Louis and Sieber, 1979), and have 
argued that developi^ig central control over the actual delivery of services 
IS extremely difficult when the project management is located m a central 
office, v^iie the actual service delivery personfiel are b^sed in the field. 
Finally, an "ev olut iona> /' imagery of program implementation argues that 
programs are cont muouslv altered in the field, so that the visible impact 
of design featuies becomes insignificant after a period of sequential 
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reinterpretat ion and modification (Farrar et al . , 1980). Thus, one of the 
questions that should be asked about any strategy for school improvement is 
whether the designers and managers of a program and the prbjects within it 
had any impact upon the services delivered to schools, and on their outcomes. 
This issue has been addressed in detai in anuthei volime (Louis and Rosen- 
blun, 1981) largely through the use of case studies of four of the seven RDU 
projects. In this section we summarize some of the main findings from 
that volune on the question of project impact on service delivery. 

Fable 9-3 presents the resuTts of several analyses of variance 
which determined whether there w<is a significant difference between all seven 
projects on variables describing the products that were used and the external 
assistance that was provide^. As is immediatel y .apparent , of 11 ANOVAs that 
were calculated^ eight resulted in significant differences between projects. 
In all cases, th.^se differences were substantial, with f statistics signifi- 
cant at the .01 level or better (not tabled). Furthermore, the character- 
istics of the intervention on which there were significant differences are 
those that were found to be hignly predictive of school improvement outcomes 
(as reported in Chapters b and 6). ' 

In addition, it is also clear from the dichotomieat ion of scores in 
the intervention variables thai there are some '*pat terns to these differences 
between projects. Two of the projects (Number 4 and Num^r 6) rank above the 
mean on all of the intervention variables for which there are significant 
differences. Another two (Project 5 and 7) rank below the mean on all 
variables. The ♦ r ema i n ing three projects have more mixed distribution^. 
Project Number 3, for example, ranked 'high on three of the four measures of 
product quality, and high on both measures of field agent services, ^it 
ranked low, however, cn the amount of training provided, both in total ^amount 
and variety. Project 1 had, m general j a profile indicating only a few 
areas of major st rengt h--the field agents spent a lot of t ime on site, and 
the products generally required considerable effort to implement. For all 
other areas, however, its profile was low. Project 2, on the other hand,^ 
appeared to promote high quality, validated products of a simpler nature, and 
tended to provide a fair amount of local technical assi si ance , ' but with less 
intensive field agent -site contact. 

Not surprisingly, these patterns of differences by project are 
highly consrstent with the distribution patterns of school outcomes. The 
d ichot om I zat icn of scores on school improvement outromei^ for the seven 
projects (also shown in Table 9-3) parallels veiy closely the high/low 
dichotomy of intervention variables. Thus, Projects 4 and 6 not only rank 
above the mean on the characteristics of the intervention, but aiso consis- 
tently rank high on school outcomes. Similarly, Projects 3 and 7 rank low on 
both intervention strategies and impacts. fhis analysis provides st i ong 
support to the hypothesis that project design features ran have a lelatively 
brocid impact upon the actual services deliverec and then outcomes. 

Bearing in mind that each RDU project was de'siqned as an interogani- 
/at lonal network, with its own network design and management characteristics 
V see .Louis and Rosenblum, 1981), we can ask ar; additjonal question. Are 
these differences in project profile a consequence of urganizat lonal design 
and management, or simply an artifact of localized adaptations within the 
project'^ lable 9-4 provides a partial answer, as it presents tf^e results 



Table 9*3 

Results of Analysis of Vsriance of Msssures 
of Chsrscteristics of the Products and External Process 
and Outcones for the Seven Projects* 



Projects 



Product Variables 


(1) 


(2) 


'(3) 


(4) (5) 


(6). 


(7) 


Product Quality (Nrl79) 


L 


H 


H 


H , L 


H 


L 


Difficulty of 
Implementation 

(Nrl79) 


H 


L 


L 


H L 


H 


L 


Field test/validation 
status (Nr90} 


L 


K 


H 


H L 

i 


H 


L 


Complexity (N=9U} 




L 


H 


H - L 


H 


L 


Adequacy of Guidance 
for Imf^lementation 

(N=90) 




(not 


significant) . 






Pre-Implementation 
Adaptation (N:90) 




(not 


significant) 






Post-Implementation 
Adaptation (N=90) 




(not 


significant) 






External Assistarfce Variables 












FA initiative (N=90) 
FA Time on Site (Nr90) 


L 

> 

^ H 


H 
L 


H 
H 


H L 
H L 


H 

H 


L 
L 


Amount of Training (N=179) 


L 


M 


L 


H L 


H 


' L 


Variety of Training (N=179) 


L 


H 


L 


H L 


H 


L 


Outcomes 














Organizational Impacts 


L 


H 


H L 


H 


L 


Product Incorporation 


H 


H 


L 


H L 


H 


L 


Process Incorporation 


H 


H 


L 


H L 


H 


L 


Problem Solved 


L 


L • 


L 


H L 


. H 


L 


Scope of Implementation 


L 


H 


L 


H L 


H ' 


L 


Personal Impacts 


L 


H 


H 


H L 


H 


4 


♦Project scores were 


dichotomized at 


the mean. 


H indicates 


that 


the 



project Mas above the meanf L that it was below. 
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Table 9*4 



nZlAUQHSHlP 8£nC£N PRWECT DESIGN AfC MANAGEUNT CHAHACTERISTICS 
WD m m INTERVENTION (t Statistics) 



Product Characteriat ir« 

Product Quality 
(NrI79) . 

' Difficulty of 
InpXonentat ion 

(Nsl79) 

Field test /Validation 
Status (N:90> 

Coiiplexity 

{N=90) 

Pre-Ifnple»entat ion 
Adaptation (N:90) 

Po 8 t - I*p 1 enen t a t ion 
Adaptation (NS90) 

Guidance for 
Implementation * 

(Nr90) 



Readiness ^ ^ ^ Span 



Project Character istica 
Centralisation Suppor t 



♦-J.il* 



♦4.41»» 



N.S. 



N.S. 



*2A9* 



N.S. 



N.S. 



N.S. 



N.S. 



N.S, 
N.S. 

N.S. 
N.S. 

+2.iO» 
N.S. 
N.S. 



♦2. 02* 

N.S. 

-►2.68»* 

N.S. 
N.S. 
-».2.58»» 



Pre-existing Structure ^' 
N.S. 

, N.S. 

N.S. 
-1.95* 
N.S. 
N.S. 
N.S. 



External Assistance 

FA Initiative 

(N=90) 

FA ri*r,e on Site 
(N=90) 

Amount of Training 

(N5179) 

Variety ol^Traming 
(N=if¥) 



♦2.98»» 



N.S. 



N.S. 



N.S. 



N.S. 



N.S. 
'••2/13* 

N.S. 
N.S. 



'f3.99»» 

N.S.f2] 

N.S. 
^2.68** 



-3. 04* 
N.S. 
N.S. 
N.S. 



Cl] More ready projects had less FA contact. 

12] t 3ig. at .06 ' • ' 

[3J Projects .ith pre-existing structure had lower FA initiative and less co^^lex products. 



♦ P < .05 
0 < .01 
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of a series of t tests m which product and technical assistance character- 
istics are claswfled by a series of dichotomized project characteristics* 
' These project characteristics are: 

*• Organizational readiness : the degree to which the 
participating organizations had previous experience 
m delivery services of the type provided m RDU; 

Pr oject span ; whether the project was organized ta serve 
a single state, or was organized on a regional/natiDnal 
basis; / , ^ 

• Project centralization ; the degree to which the central 
office of the project attempt to exercise a high level of 
influence over activities m the field; 

• Project support : the degree to which the host organi- 
zation m which the pr^oject was located w^s highly sup- 
portive of the project; and 

• Network readiness ; the degree to which the mterorgani- 

zational arrangements used m delivering services pre- ^ . 

dated the preparation of the RDU propasal. 

Table 9-4 supports the assumption thai network design features are highly 
associated with the' actual services delivered to schools, Fdr example, the' 
t. tests for project readiness are significant m nine of the 11 service 
characteristics measured. For support, significance is achieved in six of 11 

•t tests, and more scattered findings are located for span, centraJ ization and 
network readings. Moreover, the results of the t tests, ate highly con- 
sistent with earlier ^findings: projects that ara more ready la terms hf 
organizational capacity and experience deliver higher quality products, which 
typically require greater changes m the schools, which are more likely to be 
field tested, and which offer more guidance for ifrip lamentation. They also 
have field agents with higher levels of initiative, who spend more time on 
site, and provide more and more varied training (not tabled) — once again, 
variables which were foun^ to be highly ; edictive of school improvement 
outcomes. ProjeJ>ts covering areas larger than a state tended to offer 
products of Mwer quality^ lower field-test/validation statue, and less 
guidance for implementation, although they oid not differ significantly in 
the amount of assistance provided by field agents or trainers. Projects wifch 
strong support systems generally had higher quality products, and more, 
technical assistance, at least m thos^ areas where there were significant 
differences. Finally, projects which relied on pre-existing networks tended 
to use lower levels of field agent initiative (perhaps partly because they 
reJied on agents already m place who had other work commitments in addition 
to RDU), and also tended W deliver products of lower complexity; more 
centralized projects were more likely to support pre-implementation adapta- 
tions, and were also more likely to deliver high levels of field .agent time 

^at the site level. ^ 

Readers who are interested irt a more elaborate explanation for 
seme of these findings should turn to' Louis and Rosenblum (1981). However, 
we may summarize the data presented there as follows. First, projects have 
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important effects on the nature of services that are delivered, even m a 
demonstration project which is m an evolutionary, early implementation 
stage^ and which relies on a dispersed . organizational st.ructure with j^la- 
tively low supervisory authority. Second, it is plear that 'some features of 
organizational design, most particularly the readiness of the organiz<ations 
and the degree to which the project host ^provides high levels of^ support for 
the project, tend to be associated with the delivery of services that are 
shonn m our previous chapters to be associated. with higher levels of school 
improvement outcomes of various types. 

• . • 

These findings do not discredit theoretical and empirical studies 
which emphasize the "looseness" of mterorganizational modes of service 
deliverers, and the problems of delivering services as they are planned 
at federal and sta^e levelsP. Rather, they simply serve as a- caution to 
theorists- to refraiTT from ovennterpreting evolutionary or "loose coupling" 
explanations. While the design and management of intervention programs is 
far /"rom a science, vie may still expect to see some correspondence between 
organizat'ionar intent and action. Several of the seven projects that were 
studied intended, for example, to defemphasize field agent initiative and time 
on >site, and to emphasize simple products which did not necessarily meet 
criteria of strict validation. These' stand out, in oOr profiles in Table 
*9-3, as having done what they intended, at least to some degree. In addi- 
tion, t\)e analysis presented briefly here, and in greater detail in Louis and 
Rosenblum (1981), indicates that sound planning and management practices are 
reflected both m more consistent .delivery of services, and also in higher 
levels of school outcomes. While there was con^siderable local evolution in 
both design and management, key features of the projects' strategy of inter- 
vention tended to be preserved m ways that are measurable at the school 
level. 

THE IMPACT OF THE INTERVENTION: 
PRODUCTS, PROCESS, PEOPLE AND SITES 

The previous sections of this- chapter have attempted to explicate- 
relationships between groups of. variables which reflect the local features 
of the school site and the interventions, basically focusing on "fiJling in" 
hypothesized causal rel at lonsKips which were presente(J m a schema for 
analyzing the outcomes of school improvement efforts. In this section, the 
general theme of this chapter, which is to better understand the dynamics of 
the powerful predictive results presented m Chapters 5 through 8, will be 
maintained. However, we shift themes, and move from trying to understand how 
various aspects of the intervent)on are explained, either by. site characteris- 
tics, project characterxstics, or each other, back to '*an emphasis on the 
school improvement outcomes. Two ma^or questions are addressed in the 
remainder of the chapter.. The first focuses on how three components of the 
RDU intervention— products, process, and people to provide assistance — con- 
tribute jointly to the outcomes. The second concerns ±he deyree to which the 
RDU intervention strategies have an impact on the school improvement outc5omes 
(organizational change, program incorporation, process incorporation, probJem 
resolution ^nd staff development) over and above that which is explained by 
local site characteristics. J? 
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CcHnbminq t he Three Ir t ervention Strategies 

Previous chapters examined trfe impact of each aspect of the interven- 
tion separately (arrows 3, 6 and 7). Overall, characteristics of the product 
and of external hUman. assistance each separately explained greater percent- * 
ages of variance m school outcomes than the internal problem-solving activi- 
ties. Not surprisingly, incorporation of the problem-solving process was the 
only outcome that was affected more by the internal problem-soiving activi- 
ties than by either of the two external interventions. 

However, the impact rf the RDU « intervention cannot be Widerstood 
by only examining the three intervention strategies separately. In reality 
the intervention combined the three, strategies , and it is therefore important 
to exanfine the potency of the combined approach i.e. , arrows 5 , 6 and 7 
together). Two separat^ statistjv^al approaches were used in this explora- 
tion: canonical correlations and multiple regressions. , 

In examining the impacrt of a complicated and ambitious program like 
RDU, we muet not only face the mOlti-f^aceted nature of the intervention 
itself, but also the fact that the program's impacts may be highly diverse. 
For RDU, therefore, the disaifiuLnyly simple question, "Did it work?" does not 
have an equally sinple answer unless we can make use of an analytic tech- 
nique which allows us to simultaneously relate a number of "treatment" 
indicatory to several outcome measures m a single analytic operation. 
Canonical correlation , analysis provides us with just such a to61 by deter- 
mining whether any combination of the treatment variables (a canonical 
variable) is signifiai^tly related to any combination of outcome measures* 
(also a canonical variable), and if so, how many such combinations may be 
identified, and what is their composition? 

To make an assessment of the impact of the combined RDU int^ention 
strategy, a group of 19 treatment 'measures and e second group of five outcome 
measures were used for the canonical correlation analysis. TKese variables 
cover the gamut of product, external assistance and internal process treat- 
ment indicators, and the ultimate site-level outcomes. 

The canonical analysis revealed a single significant canonical 
correlation of .84 (p< .001). That is, the analysis identified a canonical 
variable, or best weighted combination of treatment variables which was 
significantly correlated with'^'^'ff^ombination of outcome measures, strongly 
indicating that a the treatment was, m fact, associated with the outcomes^ 
But it IS the composition of these treatmjsnt and outcome combinations which 
IS of greatest interest here. The most salient treatment and outcome varia- 
bles were identified by taking .50 as the minimum "significant" correlation 
between a treatment -or outcome measure and the canonical variable encompass^ 
mg it. These appear with asterisks m Table 9-5, where it is interesting to 
note that all three asptects of the RDU treatment ar^^repre8ented- >*-product , 
process, and external resources. - In addition, the relativfe size of the 
coefficients is similar between variable groups. This suggests that not only 
IS the RDU treatment related to outcomes, but also that the optimal treatment 
^includes all three of the RDU strategies. 

\ 
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Table 9-5 

HANONlCAk ANALYSIS OT RDU TREATfCNT WACT 



Principal Influence 

faculty Influence. 

ftreatlth of Involveaent 
in Solution Selection 

Breadth of Involvement 
4n iMpleMentation 

Level of Effort 
t 

Quality of Process 

Agent / Pr inci pa i Contact 

Agent Innovativeneas 

Agent Tuae on ^ite 

Agent Initiative 

Age ntf Political 
Perspectives 

Agent Structural 
Perapective 

Amount of [gaining 

Diversity of framing 
Sources 



Product Quality 

Difficulty of Implementation 

Product Validated 

Relative Advaniage i 

ProJuct Complexity 



TREi:>CNT fCASURES 
Internal Problem Solving Activities 
.45 • 
.51* 

* ' .51* 

.35 

External Aaaistance 

-.30 

7.10 , 

.40 

.42 
-.02 

-.09 

.50* 

Product Ciaaracteristics 
. .56* 
,.5i« 
.42 
.5i« 
.53« 



Problem Solved 
Organizetionai Impacts 
Personal Impacts 
Product Incorporation 
Process' Incorporstion 
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The combination of "aiqnificant" outcome measures making up the 
« global outcome canonical variable includea the following: 

• the extent to which principal and teachers report the 
problem has been solved; 

f impacts on, the school as an ^organization; 

• personal impacts on participating staff; and 

• incorporation of the adopted product/materials. ♦ ^ 

IncorpSration of the problem-solving process was not a salient component of 
the canonical outcome varialble ii. this analyses of the RDU program's site- 
level impacts. 

The combination of treatment measures is informative. The product 
chafacteristics which are involved include the overall assessment of the. 
product's quality drawn from the 'teacher survey* Jhis assessment includes 
ratings of whether the product was relevant to school problems, met a need in'' 
the classroom, and provided new information. A second product characteristic 
is teachers* ratings of how difficult the product was to implement. This 
rating includes such considerations as the extent to which implementation of 
the product required the teachers to change the ways their classrooms were 
organized, changes in teaching style, substantial additional record keeping, 
and a general assessment of how hard the product was to implement. The 
extent to which the adopted product or materials represented an improvement 
over previous practice was alsq "important**. Closely related to the assess- 
ment of difficulty in implementation is the other salient product variable, 
the complexity of the products < 

Note that^all four product variables are positively related to 
school outcomes. That is, not only is product quality positively related 
to outcomes but difficulty of implementation and complexity are also posi- 
tively related to program impacts. This suggests that where a greater level 
of effort is involved in implementing the product, its impact is likely to be 
greater, perhaps because it requires greater commitment on the part of its 
users. This interpretation is also supported by the other salient treatment 
variables in this analysis. Among the process indicators, for example, 
breadth of participation (in the sense of the number of different groups 
involved) in implementation activities is also positively related to school 
outcomes, as is the overall level of influence the faculty as a whqle has on 
the problem-solving process. Finally, the number of sources from which 
school staff received training in product use, and the total amount of 
trainring received were also positively related to outcomes. 

In summary, this analysis suggests that while all three dimensions of 
the RDU treatment are positively related to school-level outcomes, the 
optimal treatment may include aspects of all three in combination. 
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Multiple regressions of outcomes on a ' set of independent variables 
drawn from each of the three intervention strategies were also conducted. 
The following variables were chosen: 

• product variables : product quality, product complexity, 
product validated, and difficulty of implementation; 

• external human assistance variables : agent/principal 
contact, amount of training received, diversity of 
training, and agent time on site} and 

• internal problem-solving process variables : faculty - 
influence in the process, breadth of participation 

in solution selection, breadth of participation in 
implementation and the quality of the problem-solving 
process. • 

\ " Highly consistent with the canonical analysis. Table 9-6 indicates ' 
thai the real potency of the intervention is likely to have been a function 
of the combination of strategies, resulting m high or very high percentages 
of variant^ explained on each of the school outcomes, (for example, adjusted 
multiple R s were well over 50% for organizational impacts and for product 
incorporation. Even process incorporation, the most elgsive of the school 
outcomes in our analyses, had 2^ of the variance explained by the best 
weighted sum of six variables drawn from each of the intervention cate- 
gories.) It IS particularly interesting that, for each outcome, the vari- 
ables that contributed to the explanation of the outcome were drawn from each 
of the intervention strategies. Furthermore, with the exception of one 
outcome, a best weighted sum of intervention strategies is a 'more powerful 
predictor of the outcome than any of the individual intervention categories 
(see Table 9-7). The one exception is the outcome of "problem solved" which 
IS pred:^cted fetter by product variables (R =.46) than by a combination of 
strategies (R =,41). 

The most important predictor variables of the combined intervention 
strategies are product quality (which enters into the equation for each 
outcome); product characteristics such as complexity and prior validation, 
amouQk of training received, agent time on site, faculty influence in 
the process and breadth of participation in solution selection and imf51emen- 
tation. Product characteristics and diversity of training appear to be 
particulaily important to product incorporation; but ironically, product 
quality and prior validation are negatively related to process incorporation. 
The only significant positive association with both program outcomes occurred 
Q for the diversity of training variable. 

How can one explain the relatively low impact of the intervention 
on process incorporation'? While most .f the RQU projects had stated objec- 
tives of process incorporation, case study and site visit data reveal that 
in fact the primary focus of the intervention was to provide assistance 
for engaging in a specifically targeted problem-solving process with the 
objective of ultimately adopting and installing a new product or practice to 
solve a particular problem. Wtule some training in the process was included, 
it was hard for the sites to concentrate on the capacity-building function at 
the same time as effort •was being* expended to solve a particular problem. 
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.rabU 9-6 

STANDARDIZED STEPI^ISE RWHESSION (flCTA)^ COCrriCIENTS 
FOR THE RELATIONSHIP BETMCEN COMBINED INTERVENTION STRATEGIES 
SIX MEASURES OF SCHOa OUTCOMES 
(Ns75) . 



Intarvent ion 
Str«t09i08 



HiWurea of School Outcomi 

Organiiationai Product Process ProblM Scope of Personal 

lapscts Incorporation Incorporation Solved InpleMentation Impacts 



(Prddtj?t) 
Product Quality 
Product Coapiexity 
Product Vslidsted 
Oiifficulty of IniplBaentstion 



.18» 



.12 
.15» 

a 8* 



-.27»» 



.22» 



,20* 



.28»» 



^EiU-ernsl Assittsnce) ' 

Agent /Pr inci|]^ w Cont ac t 

Anount of Training 

Diversity of Trsining 
Sources 

Agent Tise on Site 



.22»» 



.14 



.18» 
.23» 



.11 



.21* 



.13 
.11** 

-.30^ 



Clnternsl Pcoblew^Soiving 
Activities) 

Faculty Influence 

Breadth of Involvesent 
in Solt'cion Selection 

Breadth of Involvenent 
in I1141 lament at ion 

Quality of Process^ 



>%jltiple R^ 
Adjuated Multiple R^ 



.09 



.16 



.11» 



.59 
.55 



.2i« 



.09 
.20* 



.16* 



.56 
.52 



.30 
.24 



.43 
.41 



.16 



47 
.43 



.08 
-.17 



.42 
.36 



^ 

♦ Bets Coefficients are preaently only for those vanablea which contributed to the reported multiple R^, 

The selection process n^s stopped when additions! vsnables fsiled to incresse the Multiple R by IS or.^mors; 
the order of sntry wss unforced. 

♦ p < .05 ' : ' 
p < .01 • 
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Table 9-7 

PERCENTAGE OF VARIANCE IN OUTCOMES EXPLAINED BY THREE STRATEGIES 
or THE INTERVENTION AND THE COMBINED INTERVENTION STRATEGIES* 

(N = 75) 



Predictor V 






Outcomes 








Variables 


Organizational 
Impacts 


Product 
Incorporation 


Process 
Incorporation 


f'robiem 
Solved 


Scope of 
Implementation 


Personal 
Impacts 


Product Characteristics 

External Assistance 

. Internal Problem-Solving 
' Activities 

/ 


.28 
.36 
.34 


.40 
.40 
.12 


,.10 
.10 
.15 


.46 
.17 
.11 


.26 
.41 
.12 


. .30 
.14 

. .02 


Combined Intervention 
Strategies 


.55 


.52 


.24 


.41 


•^3 


.36 



*Ad justed multiple R . 
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Furthermore, the field agent or facilitator was viewed as crucial to the 
process, and without special project support, was not likely to be available 
to the local site again. 

Since the paths for reaching the objectives of process incorporation 
and. product incorporation are quite different, it should be emphasized that 
the interpretation of their potential incompatibility is quite speculative 
and does not imply that there is any inherent conflict between the RDU 
objectives of building capacity through increasing participation and ration- 
ality in the problem-solving process and implementing a high quality program. 
When it comes to the question of incorporation, however, it may well be that 
the level of effort and commitment to routinize the use. of both product and 
proc3ss IS Loo great for most schools. Our site visits revealed, for 
example, many instances in which school district administrators, in making 
resource allocation decisions after the end of the RDU project, made implicit 
commitments to either process or product. For example, in -one district, 
where it was agreed by many staff members that the participatory process wan 
a key ingredient to the effective implementation of a complex reading program 
m the juniur high school, the district office nevertheless determined that 
the program would be implemented by fiat in the remaining junior high schools 
in the district, because they could not afford to have each school go through 
a similarly intensive period of decision making and training. In another 
district, a principal freely admitted that the use of R&D products p er se was 
of little interest to him after completing the project: what he wanted to 
preserve was the annual process of examiningi some aspect of the school's 
goals and functioning as a group.* 

The Relative Potency of Intervention and Contort 

The previous analyses have demonstrated that the combined interven- 
tion strategies that were utilized in the RDU program were more highly 
associated with school outcomes than was each strategy separately. A rnajor 
objective of this chapter remains — to examine the relative importance of the 
intervention compared with the site characteristics on the school improvement 
outcomes (arrow 5, 6 and 7 together, compared with arrow 8 in. I he model). As 
described m Chapter 8, site characteristics were strongly associated with 
school outcomes. A first step in this analysis was to compare the explana- 
tory nJtility of site characteristics and the combined intervention strate- 
gies. Table 9-8 indicates that for all but one outcome measureT~the effects 
of the intervention outweigh site characteristics in accq^ihting fgr variance 
in school outcomes. The ^iggest difference is in the adjusted R for staff 
development outcomes (R 2 -56 vs. R = .16), followed by an effect on 
organizational changes [H = .55 versus R =.40). In other words, the 
spin-off effects of the program were most markedly affected hy the interven- 
o tion. Only process incorporation was equally affected by both the interven- 
tion and the site characteristics, and in each instance only 245o of- the 
variance was explained by each category. 



♦In very few schools that we visited did w^ find administrators 
♦ who were equjally committed to preserving both the fifx)duct and the process. 
In perhaps the most extreme casp , the principal avowed that , in h is mner- 
city school, the teachers should not incorporate any innovation permanently; 
part of his approach to avoiding "burn-out" was to interject innovation 
each year. • 
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Table 9-i 



PERCENTAGE t)F VARIANCE IN OUTCOMES EXPLAINE^^BY COMBINED INTERVENTI 



AND SCHOOL CHARACTERISTICS* STRATEGIES 
(N = 43) 



Outcomes | 

Predictor , Organizational Product PmoP^Q Pr,,k»„-. p t-- n 1- 

Variables ImDart.i tll^n \ ^ process Problem Scope of Personal 

impacts Incorporatmn Incorporation Solved Implementation. Impacts' 



Combined intervention 
M Stiateqies 

School Characteristics 


.55 


.52 




.24 


.41 


.43 


• 

.36 


.40 


.45 




.24 


.34 


.34 


.16 



-^Adjusted multiple R 
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Two additional analyses were conducted to determine whether site 
characteriHi^s add to the ability of the intervention to account for school 
outcomes* ble'pwise regressions were conducted of outcome measures on 
variables representing each component of the intervention (products, external 
human assistance, and internal problem-solving activities) and potent site 
characteristics. As Table 9-9 suggests, for all outcomes, explanatory power 
IS increased when variables from all of the above components are considered. 
Eight variable's explain 68% of the variance in organizational impacts, and 
once again process incorporation is the most elusive, with 29% of the vari- 
ance explained. It is particularly interesting to note that for almos^ all 
of the outcomes, the variables contributing to the adjusted multiple R are 
drawn from all the domains of the intervention (products, external as- 
sistance, and internal process) as well as site characteristics. The excep- 
tions are the degree to which the problem was perceived as solved, in which 
nq variable representing external human assistance entered at the point in 
which the selection was made, and process incorporation, where the explana- 
tory variables represent only the internal problem-solving activities and 
site characteristics. In this analysis, nj9 variables representing, the 
exlernal product characteristics or external /iman assistance contributed to 
the explanation of process incorporation. / This analysis reinforces the 
previously stated interpretation that the intervention may not have success- 
fully fostered process incorporation. Instead, the degree to which the 
outcooip was acTfr^ed was largely a function of the internal processes (which 
were less influenced by the project than by the external features of the 
intervention), and the less manipulable site characteristics themselves. 

The final and most stringent test of the relative power of interven- 
tion and site characteristics invulved a block stepwise reoression model. 
First, a sei^ected number of site and problem context char^'.t£eristics were 
entered in an initial block--in this case, teacher change jor lent ation , 
problem focus on classroom organization, ttie index of diaadvantagement, 
principal influence in the school and problem focus on pupil performance. 
The second step allowed unforced entry of the nine most powerful intervention 
variables. The purpose of this analysis was to determine ^whether , after the 
most powerful- site and problem context explanations have "used up*' the 
variance in the dependent variable, the int^ervention effects can still make a 
significant contribution.* The results of this heuristic "test" are pre- 
sented in Table 9-10. 

The major feature of this table^ is that for all outcomes, interven- 
t ion ^^.easures increase the multiple R— significantly o\fer the contribution 
matie by five important site characteristics . Since this finding is repli- 
cated in other regressions using different combinations of external variables 



♦Because of our relatively small N, it is necessary to allow a 
limited ^number of site characteristics to enter in the first block in order 
*to ensure that"*the degrees of freedom ^ailable for some intervention impact 
are preserved. Thua, the multiple R do not correspond to some tables 
presented earlier. In order to ensure that the findings discussed here are 
not solely a functifn of the particular tnree site charactei^istics selected 
as the first block, additional combinations of site characteristics were used 
in other test runs. The findings are similar to those presented here, 
although the actual regression coefficients vary. 
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Tabu 9-9 



STANMRDUEO STEWlSC REGRESSION (BETA) COEFFICIENTS 
FOR iHE RELATIONSHIP BCTICEN COMBINED INTERVENTION STRATEGIES AND SCHOa CHARACTERISTICS AM) 

SIX MEASURES OF SCHOa OUTCOMES 
(N«%9) 



Pr«dictor 



Product Quality 
DiTTfetflty of laplMMintation 
Prtxlurt CoiN>U«ity 
Product Validated 



School Out COM! 



Orgahiiational Product ProcMt 
Iir^tjta Incorporttion Incorporation Solved 



Probles Scope of 



Personel 



,25 
.14 



Ii^)lei»entetion Impacte 



.18*» 



-.17 



.14 
.20 



.26»» 



.20 



(External Aneietance) 

Aq«nt/Pr«ncipel Contact 
Agent Tinie or Site 
J^MMint of Training 



.16» 



.16» 



.43* 



(Internal Problei-Solving 
Actuitiec) 

Faculty Influence 

Breadth of Involvement 
in Solut on Selection 

Breadth uf Involvement 
in Imp lenient at ion 



.21* 



.37« 



.20 



.25» 



.11 



.21 



.16 



.09 



(Sch(X)l Characteriatica) 

Teacher Change Orientation .3l»* 
Principal Influence .27»» 
Prob. in Pupil Perf. 
Prob. in Ciaanroon Org. .18* 
Inden of Oiaadvaotagemnt 



Multiple R 

Adjuated Multiple R^ 



.73 
.6B 



.31** 



.27* 



.67 
.63 



.19 
-.26» 



.>5 
.29 



.23 



.37* 



.59 
.53 



.22* 



.60 
.53 



.30* 
.20* 



.47 
.40 



J 2 

♦ Beta Coefficienta are presently only for those variAlea which contributed to the, reported multiple R . 

The aelection oroceaa i4a atopped when additional variiblea failed to increaae the Multiple R by IX or noret 
the order of entry we* Enforced. 

\ 
p < .01 



• p < .05 



EMC 



\ 
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Tablii 9.10 



STAIOAROIZCO RCGMCSSION (BCTA)^ COCrriClCNTS TOR m RCl AT IONSHIf> 
KTWeCN sue CHARACTCKISMCS (fCWCCO ENTRY), mWllhlOH 
VARUaC$ (STCf>MISe ENTRY) AND SIX HCASURCS QT s6H00L OUTCOMES 
(N«55) » 



t 


ViriabUi 








School OutcoMea 








Organizat 
iMpacta 


lonal Product 

Incorporation 


Proceaa 

Irtcorporation 


Problea 
Solved 


Scope of 
lapleaent at ion 


Pereonal 
l«pacta 


(SchMl rhBr«ct«riitici) 
















T«art>«r Chvig^ Oi««niition 




..27» 


.2ft» 


.21 


-/J5 


.20 


.09 


Prineipil Inflmmce 




.01 


.06 


-.10 


.17 


-*17 




Prob. in Pupil P«rforMncD 




.12 


.2d»» 


• .16 


.au** 


.17 


-05 


Prob. in ClMiroom brg. 




.2I» 


.07 


-.10 


.02 


.uo 




Indm of D'ltdvantagmMnt 




.03 


.03 


.01 


. .09 


.07 


.20® 


(Product) 
















Product QuoJfity 






.18» 




.38»» 




» 

.28»« 


Product Co«plpiiity 
Product vVlidited 


>* 


.23»» 




-.17 


* 


.15 


.19 


(Cxternol Asoiotvlvf} * • 
AMxjnt or Triining 
















Aqmt Tiiif» on Site 






.12 




-.19 


.25* 


•.3«" 


Aqent/Principal Contact 
















(Internal Pro61ea-5olving>ct ivit ie« 
















farulty Influence 




.15»» 






.I2» 




.11 


Breadth of Involveevnt 
in Solution Selection 








.22 




.U 




o 

Breadth of Involvement 
in I«ple«entat ion 




.28»» 


.38»» 




.20* 


.21 




l^iUipla R^ 


.29 




.28 .01 


.23 


.18 






Adjusted Multiple R^ 


.58 




:62 .17 


.50 


.43 


.39 





♦ Beta Coefficienta are preaently only for thoae variiblea i*hich contributed' to the reported Multiple 

The aelection proceaa mm atoppml when additional variiblaa failed to incraaM the Multiple R^ by iX or aora. 

• . p < .05 

p < .01 
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(not tabled), it suggests that the^ interventions made a major contribution to 
school improvement outcomes over and above the site characteristics^ 



We .would caiition against any major interpretation of the regression 
coefficients withm this table because of the forced entry of a limited 
numbet of variables, not all of which are significant. However, it is also 
interesting to note that, while many of the coefficients are not significant, 
measures from all three components of the RDU intervention enter each regres- 
sion. This again supports our previous argument that people, process and 
produi^ts are combined to produce effective knowledge utilization and school 
improvement* 

SUMMARY AND CONCLUSION 

Because the set\of questions and findings presented ^n this chapter 
IS relatively diverse, a brief sunmary is appropriate: 

• Site characteristics and external technical assistance 

are moderately good predictors of the internal^ problem- ^ 

solving process.. Site characteristics tend to be more 

powerful ih determining the process, with the exception 

of th^ "problem-solving quality," which is strongly 

affected by the foie of the external field agent, and 

the degree to which the agent adopts an innovative 

orientation. 

• The quality and characterist^ics of the internal problem- . 
solving protfess are of limited value m explaining the 
degree to which a high quality, validated product re- 
quiring a broad scope o^ change is selected* Our 
analysis revealed, however, that insofar as the process 
was important, it appeared to be a result of the activity 
of two potential "internal change agents" — the principal 
and central office specialists* 

• The project design and management appear to account for 
•significant differences in the typeJof products selected, 

and the type of technical assist9rrt^ ser\^es delivered 
to schools* • 

• When thtf three components of the RDU intervention are 
examined together, it is the^ cc/mbmed strategy x 
rather than any efement, whiph accounts for schooX 
improvement outcVDmes* The combined strat^gjes appear 
to outweigh the importance of ^school characteristics, m 
predicting school infprovement. outcomes. 

These findings bear very directly upon the question that was posed at the 
beginning of the chapter: What is the potency of the managed approach to 
change ve^s^jg one which emphasizes the impact of "natural systefns" activities 
and other non-controllable events? There are a ntmber of observations that 
we may make, based upon the above results and findings^ emerging from 
other parts of this study (Louis, Kell, Chabotar and Sieber, 1981)* 
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First, the not^on^thdt change is managed and/or manageable must 
be distinguished from the notion that it is predictable.. We know, for 
example, that in a giverv;;1i5tri4i-year change program a variety of critical 
events wi^l occur. However, these events cannot be managed, in that the 
school administrator, 'teacher or planner cannot predict when they will occur, 
nor the ways in which they will affect an ongoing change program. Similarly, 
a principal may know that teachers* orientation to change is- one of the best 
predictors of the program's implementation and persistence — but s/he may 
have very limited ability to select staff for a positive change orientation 
under conditions of educational cutbacks, and may also have limited resources 
to attempt to alter the staff's orientation to change. S/he can, therefore, 
predict whether the school is a likely candidate for a school improvement 
program, but s/he cannot alter that aspect of the school.* 

When we examine the nature of the interventioQ and the change process 
it IS typically believed that we are dealing with issues that are open 
to management ar|d are affected by human decisions. Our data support that 
notion in fairly substantial ways, and allow us to refine some of the current 
theories about change management. For example, we can, on the basis of our 
data, go beyond the aphorism that the principal's role in the change process 
IS important. Our data suggest that th^ principal may, under typical circum- 
stances in normal, "unexceptional" schools, play a special role of quality 
control over what is adopted and implemented, ajid how its coherence and form 
are maintained in the classroom. Again, in the typical Gchool, wh^re the 
principal is not necessarily a dynamic instructional leader, the principal 
who acts 9s a "change agent" has an indirect, facilitative impact upon the 
change process'and its outcomes (see Firestone and Herriott, 1980). We also 
find that similar roles can be plliyed by central office staff members, who 
can take dn active "change agent" roles which involve facilitating more than 
decision making. 

Further, we find little evidence to support the notion that the 
world of implementation and school change is a chaotic one. Rather, our 
data suggest that, while what happens in schools may not directly correspond 
to what program designers intend, there are consistent patterns of differ- 
ences* between projects that suggest that school-based intervention programs 
can be .managed at the state an(< perhaps even federal level. 

Most important, however, is* the finding that a complex federally 
initiated and f ield-designed__pxQgram that j.s implemented at thp Inral 
school level was able to provide services which appear to at least partially 
equalize innate differences in "mnov ativeness" among schools. This suggests^ 
very strongly that, although ROU was- unqi^estiqnably a rel-tively primitivl 
tooji (particularly »yhen compared to intensive and expensive approaches to 
long-term organization development interventions, or* extensive developer- 
adopter relationshifDS as exemplified by the federally funded Follow Through 
program), some impacts m both capacity building and knowledge use can he 
achieved. 



*Even the development of contingency plans may be quite' diff icult, 
except m cases of ttae most major pifedictable events. For example, it is 
likelxvthat an administrator can estimate the probability ' of a- teachers' 
strike iTT^^any given year and decide whether s/he should iTiake contingency 
pi ansr around \this. However, it may be wasteful to develop contingency plans 
around the p6ssibility of each teacher staying or leaving. 
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We have also learned that the design of interventions to produce 
knowledge utilization and school improvement must recognize, in the delivery 
of actual services, the extensive support needs of schools. The decision- 
making and implementation process* in schools is invariably a complex one, 
which take.s a great deal of time and, if it is to work effectively, may 
require the involvement of many participants* at different stages in the 
process. While eacfi df the RDU strategies described above were /ound to 
affect school outcomes positively, it is the combinatxoffof the strategies 
that rfiost effectively predi9ts knowledge use and school improvement. The 
RDU experience- suggests that for a dissemination strategy to be effective, it 
IS important to support mari> activities? \he development of innovative, 
validated products in a^wide variety of curriculum areas that are well 
packaged, transport^le and incorporate training assistance; the support 
of reasonably intensive external hgman assistants to initiate and faciliate 
problefD solv ing, and prov ide technical assistance and trainin|) ; land the 
support of local participation in the problem-solving process to ensure local 
ownership, relevance to local needs, and a potential -capacity building within 
districts^to engage in ongoing problem-solving activities. This combination 
of external intervention and internal proCl-em solving significantly increases 
successful school improvement activities at ttie local level. 

The data further suggest that the impacts of any one of the com- 
ponents of the RDU intervention xjiscusse'9 above cannot be interpreted except 
in the light of the potency of the other components of the intervention. 
Thus, for example, the finding that extensive local adaptation and local 
materials development does not promote school improvement seems to occur 
because the faculties that successfully implemented new practices went 
through ^ _deLa_i_led_problem-solving process . In this process they carefully 
clarified their real curriculum needs, were guided, by external fiel<J agents 
through a process of matc^]lng these needs to t|ie characteristics of selected 
potential innovative practices, and, once having selected S solution, were 
able to transmit their enthusiasm to tfie whole faculty. Because the solution 
acL|:ually matched a felt need reasonably well, gross adaptations were typical- 
ly hot necesary. A "sense of ownership," which is often found to beSfelated 
to incorpt)rat ion of new practices, was developed through faculty involvement 
in the decision-making process, and not through participation in local 
materials development or classroom-level adaptation. i 

In summary, in order for an R&D^ased approach to -school improvement 
ta wr1< — ^ f f e c 1 1 vfI yr rt is n e ce ssa r y to have s^yevsl m ini m a l con4i4ionfr 
0(^Curring simultaneously. First, relevant prodiicts of high quality must be 
available on a relatively easy and continuous basis. Third, the process of 
selecting and implementing a new ^practice must involve a locally driven 
scheme which is doflunated by h^gh levels of faculty involvement, strong 
sijppor t from adniin iStr ators in the school and district, and adheres at 
least minimally to principles^ of sound problem-solving. 

While our findings make a needed contribution to the understanding 
of the degree to which managed change programs carf affect schools, it is 
important nc^ to overestimate what has been reported m this chapter. Some 
of our results are disappointing. The data do nol, for. example, illuminate 
very effectively how exterr^al agencies can produce the effective internal 
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strategies or the choice of good products* We found that the external 
technical assistance provided by field agents and trainers was not especially 
po»f#erful m determining the nature of the protjslem-solvmg process used within 
the school (with the exception of the overall rationality or quality of the 
problem-solvxng behaviors). Similarly, measures of the problem-solving 
process (which, presumably, could indicate strategies for change management) 
were not very good predictors of the product choices *made by the schools. 
Thus, while we now know that certain types of technical assistance seem to 
produce desirable school outcomes, the data also provide support for ^theories 
of organizational change and decision making which suggest that the|[ manage- 
ment of behavior associated with the change process is problematic, at 
best. 

Our analysis has left us with only a partial resolution of the 
dilemma that was posed at its beginning. It has* provided clear support for 
the general cc/nclusion that a well-designed intervention can have an impact 
on school improvement and knowledge utilization i^ndependent of systeni charac- 
teristics. At the. sanie tim* , however, it is still un^lear how and why the 
potential impact ojT the int -»rvention becomes translated into a set of choices 
at the school level which, in turn, produce the desired impacts. One factor 
that we have not explored, but which is frequently looked to as a constraint 
on school behavior, is the cost of adopting and implementing a new program. 
It is to this issue that we now turn. 



CHAPTER 10 



COSTS OF PARTICIPATION IN THE RDU PROGRAM* 



INTRODUCTION 

While the emphasis of the RDU study is on the process and outcomes 
of rational problem solving m schools, it is al3o important to consider how 
much these activities cost. Every organization, whether privately owned or\ 
governmental, has lirtiited resources at its disposal. Schoois especially are 
faced with increasing fiscal constraints brought about by declining enroll- 
ments and taxpayer revolts. Thus most decisions m the RDU program and 
other .educational improvement efforts necessarily involve either implicit or 
explicit comparisons of the programs* anticipated costs to their realized or 
expected benefits. The willingness of principals and teachers to undertake' 
planned educational change is likely to be affected as much by the antici- 
pator costs^ of a proposed change as by its perceived quality, relevance, or 
repi^ ation. 

This chapter presents the results of our cost analysis of the RDU 
program. It reviews the objectives and methodology of the cost study, 
describes its data collection strategies and activities, and analyzes the 
resultant cost information by RDU project, stage of the problem-solving 
process, and other dimensions. In addition, it exposes the relationship of 
site costs to site processes and outcomes. Finally, it suggests the signi- 
ficance of these findings withm the context of the RDU program and within 
the context of future efforts to increase the use of R&D products in schools. 



Definition of Cost 




Cost accounting, the principal concern of this chapter, can be broad- 
ly-defined as the process of determining the cost of a product, a service or 
a program. But what is a xrost? An error commonly made by educators and 
others is to confuse "expenditures'* with ^coEts** and thus erroneously view 
total or per unit expenditures as equivalent^to total or per unit costs of .a 
product or service. However, expenditures reflect only budgeted or actual 
cash di^J^sements fpr specific items that the school or other organization 
needs to operate (e.g., personnel time, equipment, travel, etc.). They are 
essentially a record of input that can be uncovered by a review of budgeting 
or accounting records at the school or district level. 

Costs, on the other hand, are primar:^ly concerned with output or with 
the personnel and nonpersonnel resources actually used m providing a service 
or supporting a function (e.g., group ^brainstorming, materials development,, 
research, etc.). Costs are also more inclusive than expenditures since they 
include not only direct cash outlay but also m-kmd and indirect costs: 

• Dire^ costs can be readily identified with specific 
RDU activities and paid for directly with RDU grant 
funds; 

» . 



• ♦This chapter was written by Kent John Chabotar and Jane Sjogren. 




• In-kind costs cbn be readily identified with specific 
ROU activities but are* not charged to an RDU grant. 
In-kind cost? are incurred when Title IV-C, NDN, dis- 
trict accounts, qnd other non-RDU sources provide » re- 
sources to the RDU effort, or when teacherd and admin- 
istrators contribute their t.ime to RDU without being 
compensated directly b'y'RDU program funds;, anci ^ 

4 

• Indirect eosts are incurred for an RDU activity but 
are of a type that cannot be readily id^rjjtified with' 
the specific activity, e.g., the district's accounting 
office maintains financial records which are used by> 
the RDU program as well as by other administrative, 
instructional, and support programs. These costs may 

or may not be charged to the RDU grant. * 
%• 

By this definition, costs represent the value of all resources 
actually used, including those which are not usually reflected in granb 
awards or organizational budgets. Th'usJ thi& analysis of RDU costs entails 
not only an accumulation of regularly documented expenditures or budgets but 
also a painstaking search for less accessible cost data through interviews 
with participants, reviews of projed or site* files, and other sources. 
* 

We use this broad definition of costs not only because xt describes 
"real" costs in terms of resource use, but also because other studies, of* 
change in educational organizations, such as Bjerman and McLaughlin (1977), 
note the importance of "hidden" operating costs and their impact on the . 
maintenance of federally sponsored change. 

Objectives of the Cost Analysis 

Applying this definition of costs to the RDU program not only iden- 
tifies the "real" costs of the program, both hidden ^nd visible, but also 
provides information which can be used to assist practitioners whe^h they 
compare the anticipated costs and benefits of .alternative school iffiprovement 
strategies. In addition, it can inform the program's sponsors about how 
federal funds were .used in participating schools and whether these funds 
stimulated any local contributions of time or money to the RDU effort. With 
respect to other aspects of the RDU study, cost may explain some of thp 
variation in program impact, institutionalization of the R&D prodtict and 
problem-solving process, and other measures of RDU program "success." In 
light of these considerations, the cost study was designed to respond to tfie 
following questions: " ^ 

• What types and amounts of fe.deral and local resources 
are used at the site level and how do these vary by 
project? 

• How are costs related to the various processes threugh 

which innovation occurs, e.g., problem identification, ' ^ 
group brainstorming, etc.? 
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• \<hat use of in-kind or indirect resources is made 
at the site level and what policy implications does 
the use of in-kind resources have? * ^ 

• What information about resource use and costs may be 
most useful to state and local practitioners when 
planning for innovation? 

To answer these questions, we collected data about different types of 
costs at various levels of the RDU program with an emphasis on site*level 
costs. First, costs are divided between personnel costs (e.g., teacher time, 
consultant time, etc.) and nonpersonnel costs (e.g., R&D products, travel, 
audio-visual equipment, etcfi Second, personnel and nonpersonnel costs are 
further disaggregated into direct , in-kind , and indirect costs. 

Once identified, these six types of costs are distributed among the 
stages of the RDU problem-solving procebs (i.e., problem identification, 
solution selection, and planning for i<oiplementat ion/implementation) and 
within eabh stage among specific RDU activities (e.g., group brainstorming, 
training, administration, research, and materials development). Site-level 
costs are then grouped by RDU project in order to determine the impact on 
costs of different approaches to knowledge utilization and problem solving in 
schools. Finally, cost information derived from a site-level analysis must 
be added to costs identified through separate analyses of field agent activi- 
ties, project-level tasks, and NIE's overall responsibility for RDU program 
management. This enables us to obtain a rough "bottom line" cost for the 
entire RDU program . The relationships among these "cost centers" are depicted 
if> F igure 19-1 . . . . 

METHODOLOGY 

Like the RDU program itself, the cost study had to make the most 
efficient use possible of limited time and money to meet its objectives. 
Certain priorities were established at the start which focused the study on 
personnel costs at the site level. Because the RDU problem-solving process 
is characterized by the participation of teachers, administrators . and other 
personnel, the distribution of personnel time is the single most important 
factor in determining the composition and magnitude of costs at the site 
level. Particular attention is focused on site-level costs because this is 
ultimately where the implementation of R&D products and the problem-solving 
process, takes place. Personnel and nonpersonnel costs at the NIE, project, 
and field agent levels were also collected. These are based on readily 
available records rather than on the intensive interviews and searcihes that 
characterized the site-level analysis. Thus, our discussion of methodology 
is divided into two sections: site-level costs and other costs. * 

» Site-Level Costs * ^ 

In order to estimate site costs and to make data collection efficient 
and reasonable in scope, we devised a multi-phase procedure. The objective 
was to use full cost data collected from a sample of sites to develop stand* 
ard ratios for personnel and nonpersonnel costs and for direct and in-kind 
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FIgura 1(K1 
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costs. These ratios could the.i be used to extrapolate full costs for all 
sites based on the information on personnel use collected by the survey of 
principals. The phases were as follows. 

Phase 1 . For the first phase, detailed data about all types of costs 
(personnel and nonpersoPinel as well as direct, in-kind, and indirect) were 
collected in three generally representative sites. In February-March 1979, 
thr*e sites were visited that offered a range of projects and types of 
services (basic skills, career education,, and inservice), that had been 
visited previously by AAI staff, and that had completely implemented their 
selected R&D products. At each site, the project coordinator, teachers, and 
other school or district staff were interviewed in order to identify the 
distribution and use of specific types of resources during the course of the 
site*s RDU participation. Personnel costs were determined by the staff 
member'^ annual salary multiplied ' by the perce/itage FTE (full time equiva- 
lent) devotfed to RDU stages and activities. Indirect costs were calculated 
using the site's audited indirect cost rate as established by its state 
department of education. All other cost?s were derived from site records. 
From t/^ese data, preliminary estimates of the ratios between d irect/in-kind 
costs and personnel/nonpersonnel costs were developed. The data collected 
revealed that personnel costs accounted for over 805o of total costs. This 
affirmed the relative importance of personnel costs and also allowed a focus 
on personnel costs in the second phase. 

Phase 2 . The second phase of data collection was based on site 
visits to 23 sites in the Winter of 1980. Interview agendas were used to 
^collect somewhat less detailed information about various forms of resources 
used*, especially personnel resources, and this information was verified frc • 
project records. Data from these sites (representing all seven RDU projects) 
weri used to form estimates of site-level costs and to develop ratios of 
personnel/nonpersonnel costs and direct/in-kind costs, as well as estimates 
of total costs by site.* 



♦A third phase involved a question on the Principal Survey received 
by all principals in the RDU program. It focused on personnel resources used 
over the duration of the school's RDU involvement (i.e., how many person days 
were spent by part ic ipants on each RDU stage). This irTormation and the 
ratios developed in the preceding phase were supposed to be used to form cost 
estimates for a^l sites. For example, a 5 to 1 ratio of personnel to non- 
personnel ^derived in Phasf* 2 would allow the extrapolation* of $5 worth of 
personnel time reported in the Phase 3 survey to include an additional *i in 
nonpersonneT^costs . 

However, the cost analysis in this chapter had to be confined to the 
23 sites in Phase 2 because both the response rate and the qual ity o f the 
data from the survey were disappoint ing . The response rate on the cost 
question. |for complete and usable data) was only 275o, although the overall 
questionnaire response was 765o. In addition, there were instances in which 
the time estimates noted in the survey were significantly different from the 
estimates gathered by our staff during the Phase 2 site visits to the same 
sites. There may be two reasons for the disappointing quality of these data: 
(1) high principal turnover meant that the survey respondent may not have been 
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The cost study emphasizes site-level costs. However, in an attempt 
to estimate a rough "bottom line" cost for the entire RDU program, the costs 
of RDU <^tivities at other levels have also been estimated, 

• Field agent costs were confined to personnel costh 
based on average salary figures supplied by NIE and 
SFTE time commitments reported by field agents, 

• Project management costs were the seven projects* 
NIE grants minus funds distributed to the sites 
and field agents plus an allowanre of lOS for in- 
kind contributions. 

• NIE management costs were reported by NIE for per- 
sonnel time and travel. 

This information makes it possible to aggregate cost data across 
levels to estimate total RDU program co6ts with the understanding that 
si'ch an estimate has several limitationr First, while the site-level 
anoJysis inclixjes all types of costs, the analysis at other levels involves 
mainly personnel costs. Sejcond, th aggregation of four separate cost 
studies Hithin the same program necessarily risks some double-cGunt ing of 
fe-orgi funds which can or.ly be avoided by an expensive and time-consuning 
effo*i- to develop and apply mutually pxclusive cost categories. Third, any 
total rost figure pertains mainly ^to the RDU program since many of the costs 
revealed in the analysis are idiosyncratic of the ways in which the seven RDU 
projects were developed and operated. 

DATA COLLECTION AND ANALYSIS AT THE SITE LEVEL 

In concert with regular data collection activities at the site level 
for the overall RDU study, three set^of activities were undertaken to 
coilert site-level cost data. These activities occurred before, during and 
after the vi'^its to each site. i 

Pi lor to the site visits, the principal or another "principal in- 
formant" was informed of the purpose and nature of the visit, and was asked 
to collect information on major RDU activities and participant^. Interview 
time was requested with selected participants to verify the principal in- 



at the school for a'^ or part of *^i>e RDU program (some respondents noted that 
this was the case] and (2) the Qpst item which requested a fairly detailed 
' es^xinbe and familiarity with the operation of the project was the last item 
on the survey, and thus respondent fatigue may account for the haphazard oi* 
missing responses to this item. Because of these considerations, the survey 
dat a were used to supplement the cost data from the site visits instead of 
relying on the survey as heavily as originally planned. The site visit data 
were much more reliable since they were collected by trained researchers, 
were distributed among all seven projects, and had, in most instances, been 
verified by the project files. 



ERLC 



200 

218 



(ormant's recollections, and with the school district's fiscal officer to 
determine direct nonpersonnel and ir^direct casts of the RDU effort at the 
site. 

As the first respondent, the principal informant conveyed informa- 
tion on the range of costs incurred by the site during \ts RDU involvement. 
For personnel costs, the informant was asked about: (1) length of each RDU 
stage; (2) participants i.n each stage and their average salaries; (3) nunber 
of days m each participant's work-year so that a daily rate could be estab- 
lished; and (4) nunber of person-days expended by each participant on RDU 
activities (e.g., materials development, training, etc.) during each stage. 
Subsequent interviews with selected participants permitted refinements in 
the person-day estimates originally made by the principal informant. Using 
the dates of each RDU stage, the district fiscal officer provided the major • 
expenses during each period for RDU-related travel, RAD products, release 
time, equipment rental, and other items. The officer also noted whether or 
not an expense was charged to the RDU grant, t.ius enabling us to distinguish 
between direct and m-kmd costs. With respect to indirect costs, it was 
discovered that audited indirect cost rates existed for all sites (generally 
2-4S of direct personnel costs) which enabled us to avoid the manual calcula- 
tion of indirect costs. 

The detailed data collected through the site visits and project 
files were analyzed in order to establish full costs for each site which 
would later be combined with cost data from other sites to establish average 
costs by oroject for RDU activities and stages. Estimating nonpersonnel costs 
was fairly straightforward since the date and amount of each expenditure and 
the dates of each stage had been determined during the site visit. Indirect 
costs were also easy to estimate by applying the audited indirect cost rate. 
The process for determining direct and m-kmd personnel costs was more 
complicated, as indicated m Jable 10-1. Costs are a function of the partici- 
pants' daily rates (e.g., $100) multiplied by the nunber of person-days 
invested m each stage (e.g., 5) which yields full costs for that stage (e.g., 
$500). Full costs less direct costs charged to the RDU grant equals in-kind 
costs, most of which consisted of uncompensated staff time spent on RDU 
activities. 

COST RESULTS 

Cost data are presented for three levels: (1) site costs; (2) other 
costs; and (3) total RDU program costs. Primary attention W3s given ta* 
site-level costs because these data constitute the original focus of the cost 
study. Ir addition, they are the most reliable because they were collected 
at the site by trained project staff. Other costs are the personnel costs 
of the field agents and the management costs of the projects and NIE. 
All of these are i^-^luded m the estimation of total program costs. 

Site-L^vel Cost 

Site-level costs of the RDU project included: average total site 
costs by project and stage; average ratio of m-kmd to direct costs by 
project; and average ratio of personnel to nonpersonnel costs by project. 



219 



TABLE 10-1 

A 8ampl« Calculation of SItt PtrtoniMl Coats 



PROBLEM IDiHTlFICATION 



77 



11 



Tf 



MofittmrMr 



to 



MonihnrMr 



SOLUTION SFLECnON 



12 



7; 



70 



Month/YMr 



to 



Montti/Y««r 



SOLUTION IMPLEMENTATION' 



78 



79 



Mofitti/YMr 



to 



MofithOToar 



riAFF TYPE 



AVIfUQi 
ANNUAL 
SAURY 



DAYS 
MVOftKEO 

YEAR 



DAILY 
MTE 



9 « 



1 



40 
C 



I 



11 



I 2 

ql 2 



c c 

II 

B ^ 

w c 

-£ O 

CD 
a 



^1 
It 

I ? 
IS 



UJ ^ 





4) 












?l 




1^ 
2 « 


c 




« c 


|l 


li 






s )i 


m S 


O (0 


1- 


2 2 



S c 
|l 

UJ o 



1 
1 

Ul 



Superintendent 



50.000 



250 



200 



($200)** 



($200) 



($200)* • 



600 



600 



Principal 



25.000 



220 



115 



($460) 



($460) 



6 ($1035) 



1955 



1956 



Teachers 



18.000 



180 



100 



10 



5 

($1700) 



($1700) 



30 



10 

($5200) 



10 



8600 



1100 



7500 



Consuttar.ts 



NA 



NA 



150 



5 

($750) 



750 



750 



Other: 



$2360 



$2360 



$7185 



$11,905 



$1860 $10,066 



'Includes planning for Imptementation 
*R«ta K^sonH^ayt « cost by stage 
$200 K 1 - S200 
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In addition, average total site costs and the number of sites m 
each project can be used to estimate a tot^l site cost for that fjf^ject. 
These totals are carried for\i(ard into the next section on other costs as 
a rodgh "bottom line" for the entire RDU program is formulated*. 

Average Total Site Costs by Project and Stage 

Table 10-2 displays site costs for each RDU project and reports 
them by HDU stajje (Stage 1 - Problem Identification; Stage* 2 - Solution 
Selection; Stage 3 - Planning and Implementation). It alSo indicates the 
approximate amount of calendar timc/Vor e&ch stage.* The dollar figures 
represent the estimated total costs for all resources (personnel and non- 
personnel) used m the individual sites. (In other words, they include both 
direct and m-kind costs.) Duration of the projects ranged from approxima- 
tely^ 18 to 33 months, and averaged about 23 months. There was a fairly wide 
range m the total average estimated costs, from approximately $12,000 m the 
NEA sites to approximately $39,000 irr Georgia. These extremes are not 
surprising given NEA*s limited focus on inservice programs and Georgia's 
emphasis on local action teams of 30 ur more members which consumed large 
amoimts of personnel time. 

In nearly *all the projects, planning and implementntion activities 
took the largest amount of time and represented the largest share of total 
costs. This emphasis was particularly true at NRC and NETWORK sites which 
spent an average of about 77% of their resources on planning and implementa- 
tion, due largely to product expenses and broad staff participation in group 
brainstorming and materials development. Solution selection was usually the 
shortest and least costly stage, , sometimes because the site already had a 
product m mind when it entered the RDU program, but more often because 
a field agent or principal successfully advocated a particular product 
before a full search had been made. Pennsylvania and Michigan had unusually 
high resource use during their problem identification stage. This can be 
explained m part by a complex problem-solving process m the former case, 
and by a lack of external support m the latter case as a result of field 
agent work overload and late project start-up. 

Average Ratio of In-Kmd to Direct Costs by Project 

Comparisons between m-kmd and direct costs suggest the level 
of commitment that a site has to the RDU program. This is true because 
m-kmd costs represent local contributions of time or money while direct 
costs are charged against the RDU grant. Table 10-3 presents the average 
ratios of m-kmd to direct costs across the sites of each RDU project. 

The table reveals that in-kmd costs were a much larger share of 
estimated total costs than were direct costs. For the projects as a whole, 
the average, ratio of m-kmd to direct costs was 4:1; thus in-kind costs 
accounted for SOS of total costs. In-kmd costs were incurred m many 
ways, primarily through the time spent by teachers and principals on RDU 
activities that was not covered by RDU funds (e.g., group brainstorming and 
materials development). Some release-time money was paid to participating 
teachers from the RDU grant or district funds; for the most part, however, 

] *The cut-off point for the planning and implementation stage was 

set lat one year fpllowmg product implementation or at the end date of the 
project, whichever came first. 
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Ttble 10-2 

ROU PROJECT COSTS Al SITE LEVEL:, 
AVERAGE DURATION AND COSTS BY PROJECT 



t 

PROJECT . • Mean Duration by Stage* Mean Costs by Stage* 

(Montha) (Total $) 



< ' Stage 1 Sta^e 2 Stage 3 Total Stage 1 Stage 2 Stage 3 TOTAL 



NRC 6.5 5.5 10.5 22.5 $4,005 $3,192 $24,515 $31,712 

(2 sites) ' ^. - 



Georgia 3.6 4.0 1J.3 . 20.9 3,055 8,135 18,609 29,799 

(4 sites) . 



Pennsylvania 9.5 4.5 10.0^ 24.0 7,312 3,244 17,522 28,078 

(2 sites) , . ' 



NETWORK* 9.5 4.2 19.6 33.3 2,578 3,260 21,088 26,926 

(5 sites) • ■ • ' ' 



NEA 6^2 2.0 13.0 21.2 3,884 1,393 6,857 12,134 

(3 sitesX ■ , 

'. , 

Florida 6.0 4.6 13.0 23^.^6 ,3,761 3,914 10,252 17,927 

(5 sites) 



Michigan 6.0 9.0 3.0 18.0 9,760 6,786 * 7,791 24,337 

(2 sites) 

■ ' — ' < • ' — ■ — *- — 



grand' 

MEAN X / 6.8 4.8 il.8 23.4 $4,908 $4,275 $15,233 $24,416 

(Mean S ' (29!{) (20S) (5d«) (lOOSS) (20S) (17«) (62S) (lOOS) 
of TOTAL) 

' " ' ' ' — • — r I ■ ■■■■I ■ 

^ Stage 1: Prcblcn Identification 

Stage 2: Solution Selection 

Stage 3: Solution Implementation (including planning for implement at: lun) 




Table 10-3 



AVERAGE RATIOS OF IN^KIND TO DIRECT COSTS 
* (Across Sites Within Projects) 



In-Kind to Direct 
Cost Ratio 

PROJECT ' ^ 



( In-Kind iDirect) 



NRC 


3.6:1 • 


Geargia 


4.8:1 


Pennsylvania 


7.5:1 


NETWORK- 


4.1:1 


NEA 


4.8:1 


Florida 


3.5:1 






Mich igan 


2.0:1 ■ 


Total for 




All Projects 


4.0:1 


(Grand Mean) 






the time was contributed to the progtan without charge. Other sources of 
m^-kind costs were the time spent by school or/oistrict staff on project 
filicsl management or reporting, free use of district duplicating facilities, 
provision of district funds for travel and tuition, and time apent by non- 
school personnel (especially parents) on ROU activities. Direct costs were 
largely created by charges against the RDU grant for release time and R&D 
products. 

There wer^ considerable differjences among projects m the ratios of 
in-kmd to direct costs. Michigan, for example, only generated $2.00 m 
iri-kmd costs for each $1.00 of direct costs charged to the RDU grant« This 
low ratio IS attributable m part to the exceptionally large RDU grants 
available to each Michigan site (over $6,000). This meant that although 
Michigan's m-kmd contributions were comparable to other projects (about 
$16,000), its ratio of m-kmd to direct costs was relatively Iqw. On the 
other hpnd, high ratios were reported^ by NEA (4.8:1) and by Pennsylvania 
(7.5:1). In both cases this was mainly the result of low RDU funding per 
lie (about $2,000-$3,000) rather than unusually high commitments of local 
resources. (Another factor behind Pennsylvania's high ratio was ^the sigm-- 
f leant amount of uncompensated staff time required to undertake its complex 
decison-makmg process. ) 

It 18 notable that there was no apparent relationship between jiirect 
costs (in terms of RDU funding per site) and total site costs. Michigan's 
large RDU grants of $8,000 per site yielded an average of $24,337 in total 
costs (see Table 10-2); Pennsylvania's total costs were $28,078 even though 
its sites received an average RDU grant of only about $3,000. The relation- 
ship between direct costs and total costs is also not an inverse one — large 
RDU grants are not associated with low total costs, and vice versa. NEA, for 
ej(ampie, only allocated $2,000 per site m RDU funds and had the lowest total 
costs, avpraging $12,134 per site* 

^ f ♦ 

Average Ratio of Personnel to Nonpersonnel Costs 

Typically, personnel costs account for most of the costs of service- 
oriented programs. It was expected that up to 8S% of the total costs of the 
RDU program at the site level would be due to personnel costs. Table 10*4 
reveals that, as anticipated, the estimated value of the personnel time spent 
on project activities far outweighed the costs of nonpersonnel resource 
use. On the average, RDU sites .incurred $3.80 m personnel costs for each $1 
of nonpersonnel costs, indicating that personnel costs accounted for 79S of 
total average costs. 

Most personnel costs were due to the time spent on RDU-related 
activities by principals, teachers, and other participants. Group brain- 
storming and .materials development by local action teams prompted the largest 
time investments, although the time required for training and evaluation was 
less significant. Many sites used external consultants for training and 
technical assistance activities, adding ' another source of personnel costs. 
Nonpersonnel costs were incurred mostly for travel to observe the use of R&D 
products at other schools and for purchase of R&D products anu supplementary 
materials. 
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Table 10-.4 

AVERAGE RATIOS oKj^RSONNEL JO NONPERSONNEL COSTS 
(Across bitea Within Projects) 



PROJECT ; 


Personnel to Nonpersonnel 
Cost Ratio 


{ Personne 1 : Nbnpersonnel ) 


NRC 


5.9:1 


Georgia 


4.9:1 " - 


* 

Pennsylviania 


9.6:1 


NETWORK 


3.6:1 


NEA 


•14.0:1 


Florida 


1-9:1 


Michigan 




Total for 
All Projects 
(Grand Mean) 


3.8:1 

< 
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Although personnel time was the largest item m all sites, the ratio 
of personnel to nonpersonnel costs varied considerably across the projects. 
NEA's particularly high ratio of personnel to nonpersonnel costs (14:1) was 
m part attributable to its emphasis on mservice training. This emphasis 
ent^'ailed high personnel costs for teacher time and consultant instructional 
fees and low nonpersonnel costs due to inexpensive products and minimal 
travel to compafe alternative solutions. The complex problem-solving process 
used la Pennsylvania consumed substantial amounts of staff time and led to 
its high ratio of personnel to nonpersonnel costs (9.6:1). At the other 
extreme, the low ratao for projects m Michigan (2.2:1) is attributable m 
pare to the IdW time investments m solution implementation since Michigan 
sites tended t,o involve- fewer teachers at this stage than irv other projects 
and thus consumed less time as well. In Florida, the ratio of personnel to 
nonpersonnel costs was also low (1.9:1), largely because Florida projects 
relied on from one to three specially-trained site facilitators rather than 
on broadbased decision-making teams. This project thus used less personnel 
time overall. 

Total SiXe Costs 

To estimate the total costs incurred at the site level by project, the 
average site cost was multiplied by the number of sites m each project. For 
example, if the five sites in Project X that were included in the cost study 
reported an averse total site cost of $25,000 and Project X has a total of 
50 sites, then ,the total site cost for that project would be $1,250,000. 
Table 10-5 contains the results of applying such a formula to the RDU pro- 
jects. It also provides dollar equivalents for the ratios betw^ien (1) m-kmd 
vs. direct costs and (2) pbisonnel vs. nonpersonnel costs. 

Total site costs ranged from a high of $1.14 million in NRC to a low 
of $365,014 m Pennsylvania. These differences were due more to the nimber 
of Sites in each project than to the average cost per site; NRC and Pennsyl- 
vania had very similar average site costs ($31,712 vs. $28,078) though they 
had a wide disparity m the ntinber 6f sites served (36 vs. 13). The impact 
of a large number of sites on total site costs for the project was also 
demonstrated m Georgia and Michigan. 

The sum of the individual project totals is $5,744,544. Thi- igure 
represents the estimated total costs of ail site-level activities dcross 
all seven RDU projects. 

\ 

Other Costs 

The scope of the cost analysis can be expanded to consider costs 
incurred at otijer levels of the RDU program. This expansion is not intended 
to account for the full costs of the RDU program, but only to include major 
elements of non-site costs m estimating the "bottom line" costs of the 
program. Three non-site costs were considered: (1) field agent costs; (2) 
project management costs; and (3) NIE management costs. 
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Table 10-5 
ESTl)iATEO TOIAL SJTE COSTS BY PROJEtT 



PROXCT 
NRC 

Georgia 



Average Number Total Site 
Site of Coats by 
Coat Sites Project 



*^l,712 36 
^i,199 38 



Pennsylvania 28,078 13 



S NETWORK 
NEA 

Florida 
Michigan 

TOTALS 



26.926 29 
12,134 55 

17.927 30 
24,337 46 



$1,141,632 
1,132,362 
365,014 
780,854 
667,370 
537,810 
1,1 19, '502 



23,942 247 $5,744,544 



». In-Kind vs. Direct Costs* 
Ratio In-Kind Direct 



Personnel vs. Nonpersonnel 
Ratio Personnel Nonpersonnel 



,3.6il $ 857,713 $ 238,254 
899,642 • 187,425 
309,188 
602,636 
530,214 
401,565 
716,481 



4.8:J 
7.5:1 
4.1:1 
^.8:1 
3:5:1 
2.0:1 



41,225 
146,984 

110,461 14.0:1 
» 114,733 1.9:1 



358,241 2.2:1 



5.9:1 $ 976,178 $ 165,454 

^.9:1 940,436 191,926 

9.6:1 330,579 34,435 

3.6:1 611,103 . 169,751 

622,879 44,491 

352,358 ' 185,451 

769,658 . 349,84<1 



$4,317,439 $1,197,323 



149,844 



$4,6d8,239 ^1,136,302 



Excludes allowance for indirect costs 
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Field Agent Costs 



Estimates of field agent costs were confined to personnel, corts for 
two reasons. First, personnel costs account for up to of totai costs of 
RDU activities, including those undertaken by field agents. Second, the 
cost to the RDU study of capturing the remaining nonpersonnel costs, other 
than those at the site level, would have been exceeded by the value of 
the information, especially v^en they were a small proportion ( app/oximately 
15S) of total costs. 

Field agent personnel costs were a function of their average annual 
salaries and the number of person-years that the field agents m each project 
spent on RDU activities. NIE data were used to estimate an average field 
agent salary of about $20,000 per year (including fringe benefits). The 
number of person-years committed to the RDU program was estimated from 
data from a survey of agents m which each field agept reported the SFTE 
spent on RDU activities m a typical project year. V#ien the SFTE reported 
were summed across agents m a project and multiplied by the duration of the 
average RDU project, the approximate nunber of person-years spent by that 
project's field agents on the RDU program was obtained. This formula is 
shown below: 

Field Agent Average SSeFTE 

Personnel = Annual x reported x Duration of 

Costs Salary by Agents Project 

Table 10-6 shows the results of applying this formula to estimate 
field agent costs for the RDU projects. It reveals total agent personnel 
costs of approximately $1,626,000.' The reported nunber of person-years 
varied greatly across projects and, consequently, so did the total personnel 
costs for field agents. Projects which used full-time field agents with a 
substantial commitment to "hands-on" training and technical assistance* at the 
site levfel tended to report more person-years for their field agents than 
projects which relied on part-time agents and a knowledge base that could be 
accessed by local problem-solving teams without extensive external assistance. 

Project Management Costs 

Project management costs include the costs associated with compila- 
tion and use of the knowledge base, training and technical assistance agen- 
cies, field agent travel and other nonpersonnel expenses, project confer- 
ences, salaries of headquarters staff, and other costs involved in operating 
the project above the site level. These costs were estimated as each pro- 
ject's NIE grant, less the funds distributed to the sites (which are direct 
costs at the site level charged against the RDU/NIE grant), less the person- 
nel costs of the field agents, and plus a small allowance for the project/s 
in-kmd contributions to the RDU effort. This formula is restated below and 
was applied to the seven RDU projects. 



Project Total Direct 

Management = RDU Grant - Costs at Site - 
Costs Level 



Field Agent lOS Allowance 
Personnel for In-Kmd 

Costs Contributions 
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Table 10-6 
ESTIMATED FIELD AGENT PERSONNEL COSTS 



PROJECT 


Average 
nnnuax daiary 


Number of 
rerson Tears 


Tottfl 
rersonnei uosts 


NRC 


$20,000 


12.0 ^ 


' $ 240,000 


Georgik 4 


20,000 


14.1 


282^000 


4 

Pennsylvania 


20,000 


5.7 


114,000 


NETWORK 


20,000 


16.5 


■330,000 


NEA 


20,000- 
20,000 


7.8 


156,000 


ri^orida 


15 6 


312,000 


Michigan ^ 


20,000 f 


9.6 


192,000 




_» — 


TOTAL 


$1,626,000 
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As shown in Table 10-7, Georgia's project management costs were 
substantially lower than those in other projects, reflecting Georgia's 
smaller NIE grant, its relatively large amount of grants to sites, and its 
subrtantial field agent costs. NRC and Michigan also allocated sizeable 
portions [of ^heir NIE funds to their sites. The remaining projects used 
larger p^^t ions of their funds both ,to centralize training i \6 technical 
ass istance^unct ions and to adapt knowledge bases to their sites' needs. 
/Total oroje'^^an^gement costs were $6,081,547. 

NIE Management Costs 

The estimated total costs of the RDU projectj must include the costs 
incurred by NIE in planning, developing, iTKDni tcr ing, and closing out the 
program. The NIE-level costs began in FY 1977 and continued through FY 
1980. 

Like most of the other activities of the RDU program, NIE relied 
heavily on the use of personnel time. As shown in Tab.l- 10-8, NIE's admin- 
istrative costs of >478,600 involved the costs of a program director, moni- 
tors, a secretary, and consultants, as well as some travel costs. While 
there were some other- costs to NIE, such as overhead and support services, 
tnese were relatively minor in both size and importance. 

Unlike soTO of the ot hor o ests of the RDU progran, NIE's costs were 
purely administrative. In addition, because NIE was responsible for estab- 
lishing the program and overseeing jts operation, certain of these costs were 
"one-time" costs. That is, they would not be recurring if a program like the 
RDU program was established on a permanent basis. 



Estimated Total RDU Program Costs ^ 

A rough estimate of the tqtal costs of the RDU program includes the 
sum of costs incurred at the individual sites, field agent personnel costs, 
project management costs, and NIE management costs. In aggregating these 
costs, It IS important to note that they were incurred during the period 
between 1976 and 1979 and that the figures upon which these estimates were 
const'-ucted are based on actual resource prices during that period. As shown 
m Table 10-9 estimated total costs for the^RDU program were approximately 
$15,890,495. Federal expenditures for the IRDU program constituted 53% of 
this total or about $8,352,000. ^ 

The estimate of the total co ^ts of Ihe RDU program can be -further 
refined by subtracting the estimated costs of rest^arch and documentation 
activities at each level. The cash value of resources used in research and 
documentation was estimated through interviews with participants at all 
levels of the RDU program as well as through the review of available docu- 
ments. Adjust inc, for the costs of research and documentation is appropriate 
because theoe activities represented one-time (or non-recurring) costs. They 
were related to the information needs of Abt Associates' study and the 
earlier study by Far West Labs. Resource use for these activities was 
primarily a function of establishing and studying (rat ar than operating) the 
new federal RDU program. These modifications, elso shown in Table lG-9 
reduce the estimated total costs of the RDU program by 235© to $12,281,146. 
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Table 10-7 
ESTIMATED PROJECT OVERHEAD COSTS / 



PROJECT 


NIE-~A 


Less 




Plus 


Equals 




Grant 


Funds Distributed 
to Sites 
(Direct Costs) 
See Exhibit 10-5 


^ Field Agent 
Personnel 
Costs 
Sre Exhibit Ifl-^ 


10% Allowance 
I or in-Mna , 
Costs 


Total Project 
Management 
Costs 


NRC 


$1,256,000 


$ 238,254 


$ 240.000 


77 77^ 


i 855,521 


Georgia 


835,000 


187,425 


282,000 


36,558 ' 


* 

402,133 


Pennsylvania 


1,144,000 


41,225 


114,000. 


98,878 


1,087,653 


NETWORK 


1,421,000 


146,984 


330,000 


94,402 


1,038,418 


NEA 


1,183,000 


110,461 


156,000 


91,654 


1,008,193 


Floi Ida 


1,421,000 


114,733 


312,000 


99,427 


1,093,694 


Mich loan 


1,092,000 


358,241 


192,000 


54,176 


595,935 


TOTALS 


$8,352,000 


$1,197,323 


$1,626,000 


$552,870 


$6,081,547 
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Table 10-8 . 
NIE MANAGChCNT COSTS 



Personnel Costs 



Program Director 
Program Monitors 
Secretary 
Consultants 
Total Personnel 



2.3 person-years 
6.3 person-years 
^•0 person-years 



$ 91,200 
287,400 
48,000 
28,000 



$454,600 



Nonpersornel Costs 
Travel 

fatal fiionpersonnel 



24,000 



, 24,000 



rOTAL NIE MANAGEMENT COSTS 



$478,600 



0 
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Table 10-9 
£STIN»TED TOTAL ROU PROGRAM COSTS 



Tdtai Site dosts $ 5,744,544 

Field Agent Personnel Costs 1,626,000 

Project Hanag^nt Costs 6,081,347 

NIE Hanagement Costs 478,600 

External Research Costs^ 1,^59,804 
Abt Asbociates ($1,809,804) 

« Far West Labs i $150,000) 

Total Costs $15,890,495 
Less: 

Costs of Research and Oocunenlation: 

Site Level $ 581,456 

Field Agents 406,500 

Projects 613,729 

NIE 47,8'60 

External Research 1,959,804 



Total Research 

and Documentation Costs $ 3,609,349 



Total Costs Minus Research and $12,281,146 
Documentation Costs 
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As noted m the description of the costs sssocisted with esch of the 
levels of the progran's sctivities (sites, field sgents, projects, dnd NIE), 
most of these costs were directly relsted to the extensive use of personnel 
time. . Not only wss personnel the most heavily used form of resource in RDU 
activities, but it was slso tha single most important determinant of costs, 
m particular of m-kind costs. The service orientation of the RDU program 
activities was reflected m its labor-intensity. 

In sum, this estimate of the total costs of the entire RDU program 
includes not only expenditures made by NIE snd the seven projects but 
also the estimated value of resources such as personnel time spent on RDU- 
related activities. It includes as well the estimated total costs of all 
levels of RDU, from site-level operations to NIE administration. It must be 
emphasized that this figure represents the real cost of the program, as 
opposed to direct expenditures of federal funds. While these figures are 
estimates of the value, of all the resources utilized by the program, they do 
convey the ma^itude of the RDU program effort. 

RELATIONSHIP OF COST TO SITE-LEVEL OUTCOMES 

Another purpose of the RDU cost study was to determine the effects 
of these costs on site-level outcomes. The outcomes with which costs might 
be associated were drawn from the overall study's model for examining impacts 
of the RDU program (see Chapter 2). 

Process Outcomes 

• Incorporation of the problem-solving process , such 
as repetition of all or part of the activities and 
procedures which the process involves. 

Product Outcomes 

• tixtent to which principal and teachers report the 
problem has been solved through use of the adopted 
materials, including improvements m pupils' per- 
formance, attitudes, and behavior. 

• Incorporation of the adopted product and/or materials , 
a measure of the extent to whach use continues after 
implementation. 

UniQtended Outcomes ob Spin-offs 

• Impact s on school staff , a global measure of personal 
impacts including increased knowledge about curriculum 
development, increased self-confidence, improveo teach- 
ing skills, etc. ^. 

m Impacts on the school as an organization , a global 
measure ^of imp.ict on the school including improve- 
noents (as a result of participation m the RDU pro- 
gram) m curricLlum, materials, school organization, 
staff morale, etc. 
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To determine the relative influence of site costs on each of the 
outcomes, various types of cost were considered: (1) dollar value of total 
costs incurred by site, both for the entire project and for each stage of the 
pjroblem--solvi,ng process, (2) dollar value of direct costs and the percentage 
of total costs represented by direct costs, and (3) dollar value of in-kind 
costs and the percentage of total costs represented by m-kmd costs, 

r 

Total Cqsts 

Table 10-10 shows the results of rank order correlations between the 
outcomes and total costs. The total costs incurred at the site level during, 
the entire project were not significantly associated with any of the outcomes. 
Schools with higher total costs (both direct and m-kind) were no morp likely 
to achieve successful outcomes than schools with lower costs. Thus, it seems 
that an effort's total cost neither contributes to nor detracts from its 
success ma substantial way. However, a few significant correlations (p<.10) 
were revealed when total costs were disaggregated by stage of the RDU problem- 
solving process: problem identification, solution selection, and planning 
for implementation/implementation. There were moderate negative correlations 
between total costs incurred during problem identification and three of the 
outcomes, i.e. 'higher total costs during this stage were associated with 
lower incorporation of the problem-solving process, lower rates of problem 
solutioi^, and lower personal impacts. 

This suggests that higher costs m the early stages of a school 
improvement effort (mdst of /^hich represents the cash value of personnel 
time) may be counter-productive.- Participants may well feel overwhelmed by 
the amount of time they hs^e to invest m problem identification and become 
discouraged and "burned out" before they^even reach the solution and imple- 
mentation of a solution. This finding corroborates the earlier observation 
m Chapter 7 that high levels of participation m the problem identification 
process had little or no association with outcomes, although faculty influ- 
ence m later stages was important. 

Direct Costs 

Even if total costs were not strongly associated with outcomes, it 
was anticipated that either the dollar value or percentage contribution of 
direct or m-kmd costs might be more significant. Table 10-11 correlates 
outcomes with the dollar value of direct costs and the percentage of total 
costs represented by direct costs. It suggests that a greater reliance on 
federal funds had significantly negative results on several outcomes . Higher 
direct* costs meant lower incorporation of the problem-solving process, lower 
rates of problem solution and, most importantly, lower personai and organiz- 
ational impacts. It was observed in a few of the schools with high direct 
costs and low impacts that the dependence on the RDU grant contributed to the 
view that the RDU program was a federal experiment or intervention instead of 
a local cof,jmitment to planned school change. 



In-kmd Costs 

The percentage of m-kmd costs is more strongly associated with 
success than total m-kmd costs. Schools with a simple problem-st ' vmo 
process and low total costs^ qan still have a high percentage of m-kmd 
costs. The data m Table 10-12 suggest that a school's commitment of m- 
kmd resources either reflects or motivates a desire on the part of parti- 
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Table 10-10 

RANK ORDER CORRELATIONS BETWEEN SITE TOTAL COSTS 
AND tXITCOHES IN RDU PROGRAM / 



UUI UUflL 


TOTAL 
COSTS 
(across all 
stages) 


1 


2 

COST BY STAGE 


3 


incorporation or 
problem solving 
process (N=2i0 


on ' 

-.09 


-.48 
p<.03 


+.06 


-K.Ol 


rrooiefn solved (Zl; 


-.31 


-.37 
p<.10 


-.03 


-.24 


Incoiporation of 
R40 Product (22) 


+.20 


-.19 


-K.2$ 


+.32 


Personal impact (21) 


-.26 


-.46 
p<.03 


-.01 


-.09 


Organizational 
impact (22) 


-•-.04 


-.22 


f.06 


+.17 






• 




\ 
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Table 10-11 



RANK ORDER CORRELATIONS BETWEEN SITE DIRECT COSTS 
AND OUTCOMES IN RDU PROGRAM 



OUTCOHC 


DIRECT $ 


% DIRECT $ 


* 

Inporporation of 
problem solving 
process (N=21) 


-.28 


-.39 


Problem solved (21) 


-.46 
p<.03 


-.18 


Inctjrporation of 
R&D Product (22) 

• 


-.27 


-.21 


Personal impact (21) 

f 


. -.61 
p<.007 


-.54 
p<.01$ 


Organizational 
impact (22) 


-.40 
p<.06 


-.43 
p<.0$ 
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Table 10-12 



RANK ORDER CORRELATIONS BETWEEN SITE IN-KIND 
COSTS AND OUTCOMES IN RDU PROGRAM 



OUTCOME 



Incorporation of 
problem solving 
process (N=21) 

Problem solved (21) 



Interporafcion of 
R&D Product (22) 

Personal impact (21) 



Organizational 
impact (22) 



IN-KIND $ 




% IN-KIND $ 


+.03 




+.24 


-.17 




+.20 


+.29 




+ .41 
p<.05 


-.07 




+ .39 
p<.08 


+.18 




+ ,49 
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cipants to achieve sixcessful outcomes. Indeed, increasing levels of m- 
kind costs m the RDU program were positively and significantly associated 
with stronger organizational impacts, greater incorporation of the R&D 
product, and more tangible per9onal impacts. 

The organizational impacts of the RDU program on the school included 
a broad scope of implementation involving many students and a substantial 
portion of the class day, and major improvements m curricula and teaching 
methods. With respect to the incorporation of the R&D product, many of 
the schools reported that the product had been successfully incorporated into 
the school curriculum and guidelines had been written for the use of the 
product. Finally, personal impacts included increased confidence m teaching 
abilities, greater satisfaction in school decision making, and the acquisi- 
tion of skills m problem solving and knowledge utilization. 

Thus, m-kind costs were a significant factor m the success of 
some change efforts and the failure of others. The federal fluids expended 
by the RDU program prompted sii}stantial local investments of time and money 
which, m turn, were associated with positive school outcomes. The policy 
implication of this finding is that while extensive local contributions are 
a legitimate source of concern to grant recipients, these contributions help 
foster commitment to, and promote the success of, an externally initiated 
change program. ^ 

OVERALL FINDINGS 

These descriptions of the costs associated with the RDU program can 
be summarized as follows; 

Direct vs. In-Kmd Costs 



As noted earlier, almost all the costs associated with the RDU pro- 
gran were either direct (supported by the RDU grant) or m-kind (supported 
by non-RDU sources such as local funds. Title I or IV-C, etc.). Such a 
classification was important to estimate the full costs of RDU-related act- 
ivities at all levels of the program and to capture the "hidden" costs of 
resource use not suppdrted by RDU funds and not likely to be cited in expend- 
iture records (e.g., uicompensated staff time). This cjst study uncovered an 
extensive use of m-kmd resources for the RDU program as a whole and parti- 
cularly at the site level. 

Direct federal expenditures for the RDU program totaled approximately 
$8.4 million. This amount was allocated to the seven RDU projects who in 
turn used part of their federal funds to make small grants averaging from 
about $2,OuO to $8,000 to their sites. However, the estimated total costs 
of the entire ^DU program were approximately $15.8 million. The difference 
between these figures is primarily due to m-kind costs, or the estimated 
value of resources used which were not supported directly by federal RDU 
funds. The magnitude of this difference indicates the large extent to which 
the program as a whole relied on the use of m-kmd or "donated" resources. 

Of the three levels of the RDU program (NIE, the projects, and the 
sites), m-kmd costs were most significant at the site level. The ratio of 
direct to m-kmd costs at the site level was even lower than for the pro- 
gram as a whole, with in-kmd costs accounting for B0% of the total cus^ts of 
RDU-related activities. 



The sizeable difference between direct and m-kmd costs, particu- 
larly at the site level, highlights the extent to which federal fund expendi- 
tures underestimated the real coats of the RDU program. Indeed, much of the 
resources that went into RDU-related act iv it les— especially those at the 
site level — were supported by sources other than federal RDU funds. Both 
for the program as a whole, and especially for site-level activities, the 
use of non- federal resources iras an important ingredient m the ftOO program. 

The fact that m-kmd costs were relatively large is significant for 
planning future school improvement efforts because it reinforces the notion 
of federal grants as "seed money" rather than as the sole or permanent source 
of support, it demonstrates that even small federal grants (which m the RDU 
program were a? low as $1,000) can leverage far greater investments of local 
resources. Finally, it emphasizes the need to warn prospective participants 
that grant funds will not cover most costs^-and they must be prepared to con- 
tribute substantial amounts of their own time to meetings and other activities. 

It 13 als I important to reiterate the positive relationship between 
m-kmd costs and site-level outcomes. Successful outcomes in several areas 
were significantly correlated with higher percentages of m-kmd costs. Con- 
versely ^ total costs and direct costs often had negative relationships with 
outcomes. This implies that managers of school improvement efforts should not 
be unduly concerned with the total costs of the effort or with the amount of 
external funding it attracts. Withm the limits of the available resources, 
they should endeavor to- stimulate local contributions of time and money as a 
way not only to Support the improvement effort but also as a motivator of 
local commitment to the effort's success. 

Personnel vs. Nonpersonnel Costs 

As anticipated, personnel costs were, much larger than nonpersonnel 
costs at all levels of the RDU p^rogram. The emphasis on the use of personnel 
was not surprising, considering the amount of group brainstorming, decision 
making and training that the RDU program required. h 

A very sizeable amount of the personnel costs were in-kind rather 
than direct. That is, a great deal of the time spent by participants in RDU 
activities was not paid for with RDU funds. This was especially true at the 
site level. The time was sometimes compensated with local or other non-RDU 
finds, but was most often donated without any charge to the RDU program by 
participating principals and teachers. Again, this emphasizes the importance 
of m-kmd personnel resources m the total costs of RDU-related activities. 

Resource Use by Stage 

Of the three major stages m the RDU process (problem identification, 
solution selection, and planning and implementation), planning and implementa- 
tion activities accounted for the largest share of resource use at the site 
level. While the conditions m which the sites proceeded through the three 
stages varied, planning and implementation consistently took the most time, 
averaging about 12 months, while the first two stages averaged seven and five 
months, respectively. This was refleo^ed m greater use of resources, espec- 
ially personnel time m the third stage. Even if the amount of time for this 
stage had been limited to a period of one academic year (or nine months) as 
several program administrators suggested, the relatively heavy resource use 
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m this stage Mould continue to make it the most costly of the three stages. 
For example, planning and implementation activities m the Florida projects 
lasted an average of 13 months, >averaged $790 m costs per month, and repre- 
sented 57S of total costs. Jiv^ if the costs of the "extra" four months are 
subtracted from the tot^JrT^lanning and implementation would still represent 
51% of total costs. Mhile planning and implementation activities relied 
priman f y-tm personnel res o u r c es > the use of nonpersonnel reso^ijrces, such as 
materials or travel, was more extensive m the problem identification 
and solution selection stages. 

Another finding pertinent to resource use by stage was the negative 
relationship between total costs m the problem identification stage and 
eventual site-level outcomes. Higher total costs at *this stage meant lower 
incorporation of the problem-solving process and lower personal and organiza- 
tional impacts. Since most costs tend to be incurred for personnel time, this 
suggests that encouraging such intensive participation at an early stage of a 
school improvement effort is not an effective strategy. 

Participant Awareness of RDU Costs 

While resource use and hence estimated total cost was substantial , 
project participants at the site level were generally unaware of the extent 
to which they would need to make use of a wide variety of resources that were 
often not supplied through ROU fund expenditures. In addition, many were 
also unaware of the costs as&*7Ciated with this resource use. When asked 
about use of certain resources at specific times during the course of their 
projects, many site-level participants were able to describe their resource 
use quite readily. Hoj«*ever, when asked about the overall level of resource 
use and costs, even after the completion of their projects, many had little 
idea of the types and amounts of resources they had used and what the costs 
of the resource use had been. Indeed, both before and during project activ- 
ities, site-level personnel involved m RDU activities >iad little idea about 
their resource needs for the completion of their projects. This suggests 
that site-level staff are likely to underestimate resource needs and costs 
when planning activities similar to the RDU project. 

In addition to the sites, other levels of the RDiy program also 
possessed generally inadequate information about resource use. Project 
administration m each of the seven projects provided little guidance to the 
sites about recognizing resource needs and planning resource use. In addi- 
tion, they were aliso able to offer only limited information about their own 
resource needs and costs. For example, staff members m each of the seven 
projects were able to supply only very general information about the use of 
funds or resources from sources other than RDU funds. 

Cost Variation and , Inflation 

The costs described m this report are based on the prices of the re- 
sources used during the periods of RDU project lactivities. Thus, they rep- 
resent actual resource prices during the projects' operations from 1976 
through 1979. Although the projects took place m a wide range of locations 
(urban, suburban, and rural m many parts of the country), the prices of 
comparable resources, such as teacher and principal time, did not vary as 
widely as might have been anticipated. For example, a rural school included 
m the cost study paid its teachers an average of $12,000 per year whereas 

223 4 



244 



7^ 



a school m a relatively affluent, urban area had average teacher salaries 
slightly under $13,000* In addition, the variations seemed generally rep*- 
resentat^ve of national average prices of such items as teachers^ salaries. 

Due to price increases caused by inflation and other factors, how* 
ever, caution must be taken m applying the figures offered m this chapts"* 
to future RDU-type progtams. For example, the Consumer Price Index (CPI) 
increased from 170.5 to* 217.7* between 1976 and 1979, the period during 
ilhich the ROU projects were active; this represents an, increase of 28%. 
Even more importantly, the latest CPI (April, 1981) is 266.8 which means 
that prices haves risen another. 23S since the RDU projects ended. The 
implication of this persistent inflation is that an RDU effort which might 
have cost a school $30,000 to undertake between 1976-79 would now cost at 
least 23S more or $37,000 (with additional increases as the CPI continues 
to rise over ♦'he life of the effort). 

However, this increase m the CPI may overstate the extent of the 
increase if educational costs. For example, average teacher salaries rose 
by about ly^ during the same period. Thus, given the extensive use of per^ 
sonnel time (and of teachers m particular) in RDU activities m the sites, 
it seems likely that their annual costs may not have risen as fast as the CPI. 

Despite changes m the prices of specific resources such as teachers* 
salaries, the ratios of m-kmd/direct costs and personnel/rionperfaonne^ n^y 
be applied to cost comparisons for other years because they abstract from 
specific prices. 

Comparing RDU Costs to Other Federal Project Costs 

Several other federally sponsored programs engage ir5 activities 
roughly comparable to those of the RDU program. For' instance, Title IV-C and 
the National Diffusion Network are both federal pfograms which seek to stim- 
ulate innovation at the local school level. To offer some idea of how RDU 
costs compare to ihose of other programs, total federal expenditures for RDU 
and three other program©- are shown in Table 10-13. 

While this comparison provides only a general idea of the magnitudes 
of the four projects and of the average federal expenditures per site, it 
does indicate that the jevel of Annual federal expenditures per ROU site 
was lower than that for other projects — $11,826 for RDU compared to about 
$17,000-$23,GGG for Ti le IV-C and $27,000 for NDN.»»» 



♦CPI = 100 m 1967. 

♦♦Nat lonal Center for Educat lonal Stat ist ics, Digest of Educational 
Statistics , 1979. 

♦♦♦A more thorough audit might pinpoint ^-.ininistrative or sil)stantive 
reasons for the differences among the programs m average federal expendi- 
tures per site. However, m mid-1978, NIE and we agreed to eliminate a 
formal comparison between the RDU program and other federal dissemination 
efforts. An informal analysis might reveal that, based on the RDU cost study 
presented m this chapter^ mter-prggram cost di^'ferences are due to different 
problem-solving procedures, emphases on product development and dissemination, 
use of field agents, and other factors. 

224 

t . — ^ - 



Table 10-13 
INTER-PROGRAM COST COMPARISON 



Federal 

Federal Project Expenditures 

Expenditures Sites per Site 

fK)U Program $2,921,000 (av; per HQ** $11,826 (av 

year FY 11-19)* year)* 

Title IV-C 

Georgia 2,944,000 (FY 79) 174 .6,919 

Michigan 6,277,000 (FY 79) 274 22,910 

National 3,239,749 (FY 79) 193 27,148 

Diffusion 

Network 



♦Actual total federal expenditures averaged $2,784,000 during each 
year of the RDU program (FY 77-79) whereap federal expenditures per RDU site 
averaged $11,271 per year during this samt period. The expenditure figures 
for the RDU program cited above have been adjusted for inflation (by inflat- 
ing RDU expenditures in FY 77-78 to FY 79 prices) in order, to compare RDU 
program costs with the FY79 costs quoted for Title IV-C and NDN. 

♦♦This IS an estimate based on data from the project. In some 
cases, - project definitions of a site included multiple schools. Approxi- 
mately 300 schools received services from the program. 
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CONCLUSION 



This analysis has explored the types and levels of costs incurred 
by sites which participated in this federally-sponsored school improvemfc.t 
effort. It has presented a methodology for retrospectivly collecting data 
on the costs of various forms of resource use (e.g. personnel time, nonper- 
sonnel supplies and services, etc.) and for categorizing these costs as 
either direct costs of the RDU grant or as in-kind contributions of local 
principals, teachers, and education agencies. The study has demonstrated the 
preeminent importance of anticipating extensive personnel costs in planning 
school char^ge; these costs accounted for almost 80S of the site-level costs 
of the RDU program. In addition, it has shown that the federal funds alloca- 
ted by the RDU program leveraged large amounts of donated personnel time and 
other in-kind contributions. Each dollar of RDU grant funds prompted an 
average of four dollars of in-kind local support. 

Finally, the cost study has suggested that successful school improve- 
ment efforts do not require large investments of total or direct costs. 
Some highly successful schools that experienced a wide range of positive 
outcomes from their RDU experience incurred relatively low costs -<<hile other, 
equally successful schools, had high total or direct costs. Moreover, these 
costs did not appear to be a significant factor in less successful schools, 
either. In-kind costs were more important; the percentage of total costs 
represented by local contributions was positively correlated with several 
outcomes. In-kind costs seem to reflect or inspire a commitment to the RDU 
program and a desire to accomplish meaningful school change. This suggests 
that educators should encourage these local contributions, and focus on the 
factors Identified earlier in this report as having a significant impact on 
site-level outcomes (e.g. school characteristics, intervention strategies, 
etc.), If they seek to maximize the efficacy of school improvement efforts. 
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CHAPTER 11 
CONCLUSION AND EPILOGUE 
INTRODUCTION 

The previous ten chapters have spun out the complex ^story of how 
schools participating in a single federally funded demonstration were af- 
filed by its relatively unusual assistance strategies. Each of the analytic 
chapters includes a summary of findings. Ratheir than reiterating these; our 
intent in this chapter is to return to some of the basic issues and questions 
that were laid out in Chapter 1, and to reflect upon what has been learned 
about them. Before turning to this more speculative task, we would like to 
emphasize what we believe are the most significant of thp many findings that 
we have presented above. A more detailed discussion of the policy implica- 
tions of this study are presented elsewhere (Louis and Rosenblum, 1981) but a 
final distillation of what has been learned emphasizes the following points: 

• Dissemination programs create two type? of outcomes 
at the school level: knowledge utilization/imple- 
mentation and school improvement/capacity building; 

<? 

• Engaging *n a broad knowledge utilization actfvity 

is one of the most effective means of building capacity; 

• Good products produce goqd school outcomes: quality 
control is a critical element of an effective dissem- 
ination strategy; 



External technical assistance is important to facili- 
tate both knowledge utilization and school improve- 
ment. On the whole, training provided by experts and 
program developers that related directly to knowledge 
utilization objectives was more important than general- 
ist field agent support in producing both knowledge 
utilization and capacity building improvements; 

Field agents (generalists) were important in 
facilitating improvements in problem-solving behaviors 
related to the knowledge utilization objectives, and 
increasing the level of effort and scope of knowledge 
utilization. However, a high level of involvement by 
agents may diminish capacity-building outcomes; 

The quality of the problem-solving process is less 
important in producing knowledge utilization outcomes 
than has often been thought. How ver, it is a key 
to school improvement outcomes; 
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• School characteristics such as the staff's orientation 
to change and the amount of principal influence are 
important determinants of how well schools will im- 
plement a problem-solving process, but they do not 
overwhelm the impact of the intervention; 

• The biggest payoff m terms of both knowledge utiliz- 

ation and school improvement "will be realized by em- 
phasizing the resolution of problems that affect the 
core activities of the school — teaching and pupils; 

• Costly planned change efforts are no more liktly to have 
significant impacts on the school than less expensive 
ones. However, it is important to allocate a large 
proportion of the available resources to pav for staff 
involvement in selecting a solution and plaming for 
implementation. It is also important to supplement 
external funding witlj internally contributed staff 
time and other resources; and 

e 

• While not all schools followed program specifications 
for a rational problem-solving process, and the imple- 
mentation of an R&O-based, vpJidated "product," the 
program intervention had almost no significant nega- 
tive impacts on schools that might offset the gener- 
ally positive findings presented above. 

The remainder of this chapter is set in the framework of these basic 
findings, and they are referred to on several occasions. However, at this 
juncture we will steer a somewhat different course from the one taken in the 
detailed empirical analysis. While adhering to the realities of the find- 
ings, we will look forward to what has been learned about the process of 
change at the school level; in particular, we will look at aspects of the 
process that can be affected by externally funded but indirect interventions. 

The basic problem that motivated both the demonstration and the study 
was to learn how to create more effective strategies for disseminating 
information to schools, and for increasing the impacts of that information at 
the school level, both in terros of knowledge utilization and general school 
improvetDent . In the first chapter, we indicated that dissemination, knowl- 
edge utilization and school capacity are intertwined in many practice set- 
tings, but that they also have independent theoretical bases, and represent 
quite different sets of concerns and perspectives on innovation and planned 
change in schools. Our analysis strategy, and the summary of findings 
presented above, emphasize the strategies that may produce various change 
outcomes in schools. While this information is essential for program design 
at federal, state and local levels, the question of "dissemination for what" 
must still be considered. We will, thus, discuss some of the findings that 
can be extrapolated from the study to illuminate the dilemmas of sending 
information (dissemination) , receiving and implementing information (knowl- 
edge utilizat ion) and chanqlnq the capacity of the school to function ef- 
fectively as an educational institutution (school improvement). As part of 
this effort, conceptual schemas which further synthesize what we have learned 
about knowledge utilization and school improvement will be presented and 
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discussed. Finally, bec/Buse most dissemination programs are premised on the 
importance of rational planning and decision making , we will discuss some of 
the implications of our findings for this topic. 

DISSEMINATION AND SCHOOL CHANGE 

The RDU strategy was, first and foremost, a dissemination strategy. 
The program design emerged from basic questigns about how to send R&D-based 
information about basic skills and career education in ways that would make 
it more applicable to school's. Examples of the kinds of information that, the 
program designers 3nd project directors initially intended to disseminate 
include the regional laboratory developed Experience Based Career Education 
Programs, or, the basic skiills programs developed through Title III funds 
that were later validated by thq Joint Dissemination Review Panel and funded 
through the National Diffusion Network (NDN). Of course, as have also 
emphasizert, RDU attempted to initiate a locally driven problem-solving 
process. However, a significant TeaKire of the program was that this process 
could unfold only in thfe context of career education or basic* skills prob- 
lems, and could be addressed only through the use of validated information 
that was deemed of high enough quality to be included in a formal knowledge 
base. As* we have seen, white there were many exceptions to the intended 
restrictions, the fact that they were includ'^d in the program design indi- 
cates that the dissemination objectives we: ^ the essential driving force 
underlying all programmatic efforts. 

The RDU program may thus be easily contrasted with other strategies 
for school improvement that have been recently funded by the federal govern- 
ment. Existing "seed money" programs, such as Right to Read, make use of a 
kndWledge base and encourage dissemination, but do not limit participating 
schools to the use of existing "knowledge bases." Other programs, such as 
the NIE-funded demonstration known as the the "Documentation and Technical 
Assistance Program" (Miles, 1980), emphasized the importance of locally 
'developed knowledge, as well as "dissemination." Neither these programs, nor 
most other federal dissemination activities in education other than the NDN, 
place as much eifiphasis upon quality control in both developing a knowledge 
base and supporting implementation as did RDU. 

The RDU program disseminated information far more- broadly than 
its initial mandate to brine R&D-based curriculum and inservice products to 
local practitioners. Each project also developed a less well-defined but 
often quite cohesive knowledge base about the problem-solving process which 
was communicated .either directly or indirectly to participating schools. 
While this knowledge base of problem-solving practices was clearly embedded 
in a long tradition of organizational development research and theory, it was 
not required to pass through t.ny certification or validation procedure in the 
same way as the curriculum and inservice "products." Thus , on the whole 
the seven' RDU prbjects were disseminating packaged, tested curriculum materi- 
als, whose expectations and training requirements had been carefully worked 
out with previous users. With few exceptions the projects did not make use 
of similarly tested or' packaged materials to tram local staff members in 
problem-solving* skills or change ma^agemer.^ • but rejied on a two-stage 




process where field agents were trained, and these individuals then assisted 
(or sometiMs trained) school personnel.* 

Overall, this dissemination strategy had a considerable impact on 
many of the schools that were involved m the program. In particular, we saw 
that most of the schools completed a problem-solving process with reasonable 
success, adopted^* and implemented a product that they were satisfied with, and 
perceived a variety of other benefits from their involvement (se6 Chapter 4). 
In sotite senses, the story of ROD as we have presented it here sounds like a 
relatively singular talp of success. In the wake of current skepticism 
about the value of federally initiated change activities for local schools, 
the indirect dissemmation-based strategy employed by RDU worked for most of 
the schools involved. 

Why do we find that RDU had a signif icant /effect on local schools, 
.when others, such as Mann (1978), have Recently claimed that schools "seemed 
compel led- -some would say- doomed — to a drudging rediscovery ... .of the useful- 
ness of an axle stuck through a djsk"*^ , This is, m part, attributable to the 
general nature of the RDU intervention, and its combination of technical 
assistance, products, and a problem-solving process that was general ly an 
improvement over the more common local efforts. (The efficacy of the strate- 
gies employed by RDU have been extensively dealt with m the, previous chap- 
ters, piarticul^rly 5 through 7, and 9.) 

In. addition, however, we believe that part of the reason that we 
found RDU working is the broad view that we have taken of the possible 
outcomes of a dissemination effort. We might term our general approach as 
one that moves beyond implementation . The past ten years have seen signifi- 
cant attempts to unravel knotty issues related to the observation that, in 
schools, "plus ca change, plus c'est la meme chose." Much of this improved 
understanding of change dynamics in schools has focused on the explication of 
issues relating to how new programs are actually implemented in districts^ 
schools and classrooms. While this focus has been effective m the short 
run^ it has increased the degree to which "schcMl improvement" is regarded as 
equivalent to the transfer and use of a new instructional technology or 
curriculum. There is a need to redress the balance, and begin increasingly 
to. incorporate another kind of "knowledge utili2ation"--one that is not 
focused on implementing a new practice with fidelity to the, intent of the 
developer or researcher, but on the less tangible improvements in school 
functioning which are often viewed as general school improvement outcomes or 
the development of increased "capacity" withm the school. These school 
improvement or capacity building outcomes might be as si2)tle as reviving a 
cynical staff's ent^husiasm fo** adolescent education, or as significant and 
visible as a permanent revisicn of the way m which curriculum is reviewed 
and monitored by teachers at different grade levels, or an emergence of new 
leadership roles among the teaching staff. Our study did not initially 
intend to examine these capacity building/ school improvement outcomes, 
except to the extent that they were reflected m the program objective of 
improving the p^obLem-solv mg processed used irj the schools. However, 
because "spin off" effects were so visible m the schools that we visited, 

♦There is, however, no evidence to suggest that providing direct 
training to schools would have improved process incorporation. Among the 
four projects analyzed m a companion volume (Louis and Rosenblum, 1981), 
the two that provided direct and standardized training to sites did not score 
more highly on process incorporation. 
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mi were quickly led to emphasize the measurement of both knowledge utiliza- 
tion (implementation) outcomes and more general school improvement (capacity 
building) outcomes, even though these were not equally emphasized in the 
initial program d^ign. 

This decisiph had a major impact on our ability to report "success" 
at the school and program level. If, for example, the investigation had been ^ 
confi'n«;d to an emphasis on product incorporation and process incorporation as 
long-term measures of program success at the school level, the story would 
have looked quite different. As we have noted frequently throughout this 
volume , not only did process incorporation occur less frequently than other 
major outcomes,^ but the ROD strategies were also apparently less likely to 
increase 'the probabilities that it would occur. Indeed, some of our most 
potent results occur in the are^s of staff development and general improve- 
ments in the organizational environment. * 

This discussion has two irtiplications for the design and management 
of dissemination programs. Firot, it indicates that success is, in part, 
a matter of definitioa: a narrow focus on the assessment of a dissemination 
program may well produce a different set of conclusions. Second and more 
importantly, however, is the observation that dissemination programs can have 
broad impacts at the school level — impacts that can tiave further reaching 
consequences^ than a simple emphasis on implementation would imply., Dis- 
semination research has tended to focus on a detailed explication of the 
procesg of sending, as was noted in Chapter 1, and some of our conclusions 
for how best to improve this process will be explicated below. However, it 
is important to refine our^view of the consequences of sending and receiving 
information, particulaoHy if we hope to be able to show any connection 
between program and orgmizational improvements, and student outcomes. As we 
showed in Chapter 4, the resolution of deeply embedded school problems — 
particularly those of /student achievement — is as much a function of broad 
organizational improvement and staff development as of the incorporation of 
new curriculum practices. 

Knowledge Utilization 

We have noted that t)ie RDU program sought to promote two dista ict 
types of knowledge use — t,he use oj* new curriculum and inservice produces, 
and the use of current knpwle^Jo^ about effective organizational problem- 
solving and change-management practices. In the previous section it was 
emphasized that the program as a whole appeared to resolve local problems 
and improve schools to some degree. However, we may also ask whether it 
was equally successful in promoting the two types of knowledge utilization 
that it sought to sponsor. Based on our analysis, the answer to this ques- 
tion is clearly no. While the data suggest that the application of intended 
program strategies is clearly predictive of such knowledge-utilization 
outcomes as the scope of implementation of a new program and the incorpor- 
ation of that program into the school's routine activities, program features 
account for very little of the school's reported and observed plans to repeat 
a relatively detailed problem-solving process. In fact, both quantitative 
and qualitative data indicate that the program failed to make significant 
inroads upon the general tendency of school personnel to prefer home-grown 
solutions and local expertise (Miles, 1980). Despite the fact that in the 
RDU program both external products and external human assistance were highly 
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valued, instances of significant process incorporation were largely a 
function of pre-existing characteristics of the school (such as having higher 
leveJs'of teacher influence m decision making, and having engaged m previ- 
ous problem-solving activities). 

Why should this be ihe case? The answer lies, we believe, in the 
same explanation that Gross, et al. (1971) offered to explain the non-use of 
a teaching innovation at Cambire Elementary: vague expectations for behavior 
change and poor resources to support the change activities. In addition, a 
paradox of program design allowed schools to select a product for implementa- 
tion, but frequently imposed a problem-snlvmg process from the outside. 

First , m many cases the expectation of what the problem- solving 
process could and/or should constitute, and the underlying expectations 
for school and individual behavior were quite unclear. Teachers and princi- 
pals often did not understand why they were being asked to engage m certain 
activities a^(j, m more than a few cases, ^found them mear.irfgless and repeti- 
tive. For example, m some cases RDU projects required a needs assessment, 
even though an exist j.ng needs assessment (which may not have met the stan- 
dards set by the project) was reasonably \jp-to-date, or there, already was a 
great deal of local consensus aboUt the nature of the problem. While school 
personnel occasionally looked back and indicated that, m retrospect, the 
decision to conduct additional or more detailed needs assessments was appro- 
priate, a,t the time it was often irritating and confusing. As compared to 
the R&O-based products (most of which included some clear instructions^ for 
implementation, along with training assistance), materials to help local 
educators meet the pace and scope of tne problem-solving activities did not 
exist at the beginning of the project. There were also serious concerns on 
th^ part ^of involved school staff about the timing of activities, and many 
medicated that problem-definition activities were too drawn out whilfe careful 
implementation planning was abbreviated, or did not occur at all. 

Second, and .inore speculatively, the teachers (and- principals) occa- 
sionally felt that the process activities w&re imposed upon them without 
real consideration for local conditions. Because teachers and administrators 
participated very actively m choosing products, , the latter were able to 
avoid the "NIH" (not invented here) syndrome. The problem-solving process, 
on the other hand, no matter how effective it was felt to be, was typically 
viewed as being very different from previous prar*^ices, and prubably impossi- 
ble to duplicate withm the resources of the district. Only extremely 
inventive and committed staffs felt that they would complete a similar 
approach again without external assistance. The approach was not typically 
reinstated, but rather viewed as an' interesting but not particularly replic- 
able experience. Over half of the teachers surveyed, and almost a third of 
the principals, indicated that RDU had had no effect at all on "the way in 
which problems av^ solved m the school." 

Thus, as we move to the broader question of what we have learned 
about how to improve knowledge utilization at the local level, we must 
emphasize that our observations are confined to the utilization qf curriculum 
and inservice materials. Given this caveat, however, we believe that the 
data presented above allow uS to develop a reasonably good conceptual 
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model of how knowledge utilization was facilitated in the RDU program 
where knowledge utilization is defined as the sum of three of the RDU out- 
comes discussed in previous chapters: the magnitude or scope of implementa- 
tion, incorporation of t,he new curriculum or inservice materials into prac- 
tices m the classroom or other school change, and problem resolution. 

A Model for Knowledge Utilization 

A conceptual model was developed by sferching the regression tables 
presented m the above chapters, and looking for robust predictors (e.g., 
those that appeared as significant standardized regression coefficients in 
two or more equations).* This approach was necessary because many of the 
regression coefficients were unstable due to the change ih the variables w' '=»t 
were entered in different equations'. The model is presented in Figure 
11-1. 

The model assumes that scope of implementation »is an antecedent 
of both product , incorporation, and problem resolution, but is not a "suffi- 
cient** cause of either. Witnout an innovation of some magnitude, it is 
unlikely that an identified problem will be solved. In addition, chariges of 
low scope are unlikely to result in formal incorporation, perhaps because 
they involve alterations of common practice so minimally that they do npt 
warrant an overt effort to formally embrace the ''innovation." 

A first scan of the model shows one important feature: two vari- 
ables are^ predictors of all three knowledge utilization outcomes. These are 
product quality as perceived by the implementing staff members, and a problem 
focus on pupil performance and/or classroom organization. This cluster 
indicates that there is a '"rationsl^' basis to knowledge utilization among 
teachers, despite many arguments to the contrary.** All other things being 
equal, if teachers in a school define a problem that is associated with their 
core activities and eontext — teaching and pupils — and select a new program tor 
set of activities that they believa contains new and relevant materials and 
ideas, implementation and incorporation are much more likely to take place. 
This hypothesis is so strongly supported in our data that it leads us to 
question the accuracy of images of teachers* use of information that stress 
the unpredictability, intuitive behavior and need for personal reinvention 
that characterize teachers m their typical contexts (Huberman, 1980). We 
suggest; with all respect to those who advocate for **garbage can" or anarchic 
models of organizational behavior, that given the opportunity and an appro- 
priate set of structures, teachers ^who attempt to approximate rational 
problem-solving behavior will be more successful. This is not necessarily 
incompatible with an emphasis upon the craft orientation, or a reliance on 
intuitive judgments — these may, in fact^ be the only mechanisms for rapid 
day-to-day adaptation in the classroom. However, when making decisions 
about school-wide problems and school-wide innovation, the unpredictable, 
intuitive teacher can adopt a more managerial perspective. 



*Various colleagues have suggested that we test this model through 
statistical modeling procedures. However, becnuse of the large number of 
variables in the model, and the small N, path analysis of LISREL would i)e 
statistically unsldble and inappropriate. 

**Throughout this chapter, as in previous ones, the term rationality 
is used to connote organizational processes, and not individual choices or 
organizational outcomes. 
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FIQURE11-1 

A Hypothetical Model of Knowledge Utillietlon In Schools' 



U 



Product Quality (2) 
Classroom Pupil Focus (3) 
F.A. Time onStt6(5) 
AfDount of Training <3) 
Product Validation (2) 



Scope 




Problem Resolution 




I 



Product Incorporation 



Product Quality (4) 
Ciassroom/Pupil Focus 



Product Quality (3) 

Classroom/Pupil Focus (3) 

Product Complexity (2) 

Trainir^g Diversity (2) 

Broad Implementation 

. Involvement (4) 

Teacher Change Orientation(2) 
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* Number in parenthesis indicates the number of regression equations 
In which the variable exhibited a significant /egression coefficient. 
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Aside from product quality and a classroom/ pupil focus, there are 
substantial differences in the factors which contribute to different knowl- 
edge utilization outcomes. Scope of implementation is influenced most 
strongly, as we have seen in previous chapters, by external technical as- 
sistance, both from the field agent and from other trainers. As we have 
pointed out, the field agent contact helps to set and sustain a framework of 
expectations for behavior ( per^ps increasing the degree to which "ration- 
ality" in decision making occurs), stimulates a broader pxoblem definition, 
and helps to encourage and support faculty efforts to engage in more ex- 
tensive change programs. Increased training helps to reinforce this commit- 
ment to the innovation among a larger number of teachers and provides skills 
necessary for them to use new maicriaic? and concepts in their work. Again, 
such training may tend to set expectatmns for u^e which encourage broader " 
implementation, and more sustained and per/asive use. Product validation, on 
the other hand, does not cause broader icope. Rather, validated products 
tend to require more change on the part of teachers, and are often oriented 
to school-wide adoption. Thus, the selection of a validated product implies 
a coirmitment by the teachers and principal to broader and more comprehensive 
change programs. 

The incorporation of new practices is stimulated by a somewhat 
different set of unique predictors. The most important factor is the breadth 
of involvement in implementation. As we argO^^ in previous chapters, incor- 
poration IS by definition an organizational pij^ocess, for it involves deci- 
sions to require certain behaviors of teachers^ to replenish materials, to 
provide training for new recruits, to write curriculum guidelines, etc. The 
breadth of involvement at the time of implementation is, we believe, a 
surrogate for the spread of some concern and interest beyond the immediate 
implementing group, to a broader group composed of all interested parties. 
In addition, it reflects the change in status of the decision-making team or 
group from a small "facilitating team" to a more broadly based "implementing 
team" (Chapter Thus, ownership and commitment are spread to both deci- 

sion makers and users in an effective change process. 

Two additional variables reflect the complexity of the change pro- 
gram: the complexity of the product itself, and the divciFity of training 
sources. Where the change effort involves many components, and when the 
implementation plan calls for different kinds of support for these com- 
ponents, organizational processes are put into motion that are difficult to 
reverse. Thus, a set of supplementary career education materials can be left 
on the shelf ^nd ignored at any point, but a complicated program such as 
Experience Based Career Education (EBCE), which affects the core of the 
educational curriculum and structure of a school is harder tp put aside. In 
sum, once implemented a complex product is more difficult to completely undo, 
' although it can, of course, fade away with neglect like any other innovative 
practice . 

An antidote to fading away, however, is having an innovative staff. 
While the orientation of the teaching staff toward change does not seem to 
affect the early adoption and implementation behaviors of the school, it has 
a deep impact upon whether the change will endure into the future. Presuma- 
bly in less innovative schools discontinuation will occur through increasing 
resistance of teachers to new practices which are at odds with their more 
conservative approach to education. 
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Ohe notable feature of this entire model is the limited impact of the 
problem-solving process. This has been discussed in some detail in Chapter 
7, but IS highlighted here in*the conte>^t of a model of robust predictors of 
knowledge utilization. The administrative leadejrs in the school are repre- 
sented onl> indirectly, as actors who may support a broadly based implementa- 
tion process, and as facilitators of sound product selection. A partici- 
patory approach tKat em^Shasizes faculty involvement also seems to have little 
effect on knowledge utilization.^ As disturbing as this finding, may be to 
advocates of power equalization in schools, it is, of course, quite con- 
sistent with other findings which suggest that teacher and principal influ- 
ence in the school may have little to do with the adoption and implementation 
of comprehensive change (Rosenblum and Louis, 1981). ♦ As we shall see, this 
does n9t imply that participation and the problem-solving process are unim- 
portant. They do, however, appear to be weakly linked to knowledge utiliza- 
tion either directly or indirectly. This is, we believe, largely a function 
of the general eagerness of school staff members for better mousetraps: as 
long as the selection and initial presentation of a new set of practices to 
teachers does not arouse resentmeptf^nd appears to Jbe an improvement over 
existing practices, most t^ias^t^s will be willing to give it a try — particu- 
larly if they can be involved iri helping to design the implementation pro*- 
cess. (See also Gross et al., 1971.) * 

School Improvement • ^ 
. * 

We original ly intended that the study ' s focus on capacity building 
be limited to measures of the incorporation of an improved problem-solving 
process. However, as we have pointed out, our understanding of the ways 
in which RDU could improve schools and build their capacity for self-renewal 
(Miles and Lake, 1967) was e'-panded. Throughout the study, we have examined 
overall organization development results (such as improvements in curriculum 
and materials, ^climate, school organization, participation of teachers in 
decision making, colleagiality , morale, and the school's willingness to turn 
to external lesources) and per sonpl growth and staff development (learning, 
changes in self-confidence and attitudes toward work, leadership ami learning 
more about problem solving and R&D). Given our inability to explain process 
incorporation, these org Mzation development and staff development outcomes 
represent our operationc^ defir tion of school improvement and capacity 
bu ilding • 

Several observations may be made about when and how more general 
school improvement occurred in the RDU program. The first is that only 
in a few cases did capacity-building objectives supplant the knowledge 
utilization and innovation objectives which were at the core of the program. 
Pgr example, in only 10S of the sites did we find that schools were high 
achievers on capacity-building outcomes, but poor achievers in f<nowledge 
utilization outcomes. This statistical finding is confirmed by our site 



♦It IS clear that there is considerable controversy about the degree 
to which administrative support has a positive influence on implementation 
and institutionalization. Some studies show strong principal effects. 
However, an equal number show limited effects. This literature is r,eviewed 
in Rosenblum and Louis (1981). 
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visit data, whirh suggest very strongly that capacity building tends to 
0 ccur in conjunction with knowledge utilization activities, snd not instead 
of knowledge utilization , z 

Even more important, however, is the observation that knowledge 
utilization tends to facilitate qapaQity building in schools . Certainly one> 
of the mayi outcon&s for many of the RDU schools was a dramatic increase 
in professional communication which accompanied a participatory attempt to 
define a concrete problem, and to select a solution. And imp roved * communi-» 
cat ion ^leads, in a number of cases, to more concerted attempts to solve other 
"problems" and^'a more generalized concern with self-renewal. >4£achers who 
are actively involved in knowledge utilization activities are more likely to 
report pe^son4X str-.ff development t^pefits than teachers who do not pjartici- 
pate (Chapter , in part because of^their greater exposure to new' ideas ^nd 
consultants, and in part because of the leadership roles, that* they adopted 
with their peers. Similarly, the best predictors of organization development 
outcomes are the si^ope of implementation and the degree to which teachers 
report that the problem is sobed (Chaptered). 

As Oerr (1976) and others have noted, schools, hav-e not embraced 
the value of organization development, and may have some tendencies to 
resist it when attempts are made to impose it from the district level. ^ On 
the other hand, wtien teachers and principals become involved m a concrete 
knowledge ut li izat ion' act iv ity , capacity-building outcomes may occur either 
as a consequence of the significant effort that they expend^ in very task 
oriented ^problem solving, or because the effecti\Le School-wide' implementation 
of a new, locally chosen program increases the general sense of efficacy 
in the school. 

Despite this relatively positive conclusion about the congruence 
between knowledge utilization/implementation object-^ives aod* school improve- 
ment/capacity buiicling objectives, as we have noted above, major and perma- 
nent change in the process by which schools 'resolved their problems was 
typically not achieved. Far less thari half of the teachers surveyed, for 
example, felt that there had be^n improv-ements in the degree of participation 
that teachers had in decision making. Thus, the observation that knowledge 
utilization may stimulate broader school improvement and capacity building 
must not be inflated to a conclusion that RDU schools had become "self--renew- 
ing." In addition^ RDU represented 'a special set of^circumstances \n which 
externally imposed resources and expectations about how the knowledge utili- 
^zation process would occur treated an en^ironmf?nt that permitted more atten- 
tion to capacity building. No district, and probably few externally funded 
programs, will recreate the RDU structures and resources in a time of rapidly 
shrinking resources largely because staff development and other forms of 
organizational improvement tend to be v levied as frills in an era where basic 
school programs are under threat. However, staff and organization develop- 
ment may also take on new potency as teachers and administrators attempt to 
cope with additional fiscal constraints. Thus, a fnajor issue centers on what 
our analyses suggest ar^ the factors that local districts (or future federal- 
ly or state-funded programs^ should take into consideration if they wish to 
engage in organisation development and staff improvement in the context of a 
knowledge utilization activity. 
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Usinq the same procedure which generated a conceptual model of 
knowledge utilization 'Figure 11-1), variables >.hat contributed to the 
explanation of organizational improvement and staff development in several 
different regression equations were isolated . The relat ionship3 of these 
tc school improvement outcomes is presented in Figure 11-2* 

In this model two variables contribute to both organization and 
staff development: the complexity of the innovation (product) being imple- 
mented, and the amount of training received by staff in the site. The 
combination of these two variables reinfor^'^es the conviction, articulated 
above, that Ine process of implementing a significant curriculum change 
program is a major route for improved organizational and staff capacity. 
The complex program adopted within RDU tended to affect schools and teachers 
in multiple ways. Individually Guided Education (IGE), for example, involv s 
changing the entire staffing structure of the school, in addition to intro- 
ducing new curriculum elements. San Dieqo Right to Read (one uf the most 
popular reading programs for secondary schools) involves a potpourri of 
different approaches to stimulating student motivation, providing con'pensa- 
tory instruction, and integrating reatling into content area classroom in- 
struction. Implementing either of these or other complex programs necessi- 
tates changes of sufficient magnitude that they often stimulate the develop- 
ment of internal leaders among the teaching staffs, increase communication, 
require significant learning of new skills for teachers, and other factors 
that contribute to organizational and individual change. In the absence of 
training, on the other hand, schools may lack the resourras not only to 
implement, but to further develop understanding and capacity. However, 
typically most schools that implemented less complex innovations also sought 
less external training, and thus, did not benefit from the stimulation and 
the oppo'^tunity to learn and/or reinforce new skills. ' 

Looking further at the variables that predict organization develop- 
ment outcomes most convincingly however, it is clear that implementation 
alone does not produce organization development. Rather, the group oT 
variables that is strongly associated with the organizational improvement 
outcomes centers directly upon the nature of the process. Participation — 
both through higfi levels of faculty influence, and through the involvement of 
all significantly involved parties — is important, as is the quality of the 
problem-solv xng process. Thus, both the rationality of the decisions and the 
way in which decisions are reached are important. This cluster of predictive 
variables distinguished this model from the previous one which presented the 
relationship between knowledge utilizatiqn outcomes and predictors. While 
breadth of involvement in implementation emerges as important in both models, 
the knowledge utilization model gives no strong support to the hypothesis 
that the process of decision making is particularly important. Rather, it 
suggests that product characteristics and external assistance are most 
critical in knowledge utilization outcomes. The relative importance of 
climate and process is reinforced here by the fact that teacher change 
orientation, or the basic tendency of the staff to support and encourage 
innovation, may be key. 

In addition, the model which best explains school improvement /capa- 
city building ou^rciies suggests that too much reliance on external assistance 
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A HypotlMtical Moctol of School Improvamlnf and Capacity Building in Schools 



Complexity of 
the Product (3) 

Amount of 
Training (2) 




Organization 
Development 



Staff 
Development 
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Field Agent -Principal Contact (i 
Breadth of Involvement 
in Implementation (3) 
Faculty Involvement (3) 
Quality of the Process (2) 
Teacher Change Orientation (3) 

Field Agent Time on Site (3) 

(negative relationship) 
Classroom/Pupil Focus (2) 
Disadvantaged Pupils (2) 
Product Quality (4) 
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may be counterproductive. The association between pr mcipal-field agent 
contact and organization development outcomes, coupled with the negative 
relationship between field agent time spent on site with the team and staff 
development outcomes (Chapter 6) suggests that external assistance roles must 
be relatively circumscribed if they are to be effective m promoting capacj^ty 
building. Working through an internal agent —often the principal — may be 
more important than providing direct stimulus and support to a team. As we 
pointed out m Chapter 6, it is entirely possible that better trained' and 
more experienced agents may not have produced this counterproductive result. 
However, some support for our findings may be founr* m the Miles et al. 
(1978) stuuy of OD m schools. In that study, the involvement of external 
consultants was found to be negatively related to institutionalization. The 
authors imply that the role of external OD specialists is most effective when 
providing support to an internal OD consultant. In addition, our own study 
indicates that there is a consistent negative relationship between federal 
contribL* lons and capacity buildmg/schooi improvement outcomes. 

The distinctive factors that affect staff development ^re more 
similar to those that affect the knowledge utilization outcomes discussed 
under the previous mod^l. That is, a classroom/pupil focu§ and high product 
quality are important to achieving high levels of individual and personal 
rewards. One additional factor is of interest here, however. Schools with 
a high proportion of disadvantaged pupils are more likely to achieve staff 
development payoffs. This is probably true because morale and sense of 
efficacy may tend to be lower m these schools than m more affluent environ- 
ments, and successfully completing a problem-solving process may therefore 
have greater impacts on job satisfaction and morale. 

SOME REFLECTION ON RATIONAL PROBLEM SOLVING IN SCHOOL 

As we have noted m previous chapters, there is some consensus in 
the organizational literature that sound decision making and planning re- 
quires attention to certain aspects of the process, such as care.ul identifi- 
cation of organizational needs or problems, careful and precise definition of 
problems and what may be required to solve them, the development of consensus 
among those affected by the problem about what would be needed to solve it, a 
careful search for a variety of different solutions, and so forth. On the 
other hand, it is acknowledged that these planning activities often lack 
essential features that might be termed "rational"; they are arbitrary, 
rushed, rely on implementing "solutions" without an adequate understanding 
of what the problem is, and fall short m their adequacy of plans, blueprints 
and development of common understandings of what is to be achieved. 

Current theories m education tend to emphasize the non-rational 
aspects of decision making and planning m schools. Many theorists have gone 
so far as to question not only whether organizational rationality is actually 
the bahavior mode m schools, but whether it is possible or desirable (March 
and Cohen, 1976; Weick, 1976; Farrar et al . , 1980; Huberman, 1981). Among 
the many reasons that are presented for lack of rationality m the typical 
decision making behavior m schools are: 

• The segmented nature of work io schools, which prevents 
teachers from having sustained professional contact 
With their colleagues; 
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• The press of daily routines in the classroom, which 
leaves teachers and administrators with little time 
for more long-range planning; 

• The "craft" orientation of educators, which results 
in placing greater value on experiential knowledge 
than scientifically based or "validated" information 
from external sources; and - : - 

• The vulnerability of schools to "political" in- 
fluence from various significant outside consti- 
tuencies. 

These and other features of schools and educators are, perhaps, real 
constraints on rational behavior and decision making in schools (although we 
dcJubt that schools are, in fact, lesfs rational than other formal organiza- 
tions). In fact, many who have observed attempts at planned innovation in 
schools have noted that what is pointed tn as rational planning upon closer 
observatic?n appears to be a veneer of organizational myths which thinly 
cdvers a more richly textured but imperfect accumulation of ad hoc decisions 
and personal adaptations. 

When we began the study of school improvement in the schools served 
by ROU, we too expected to find little evidence) ^ of rational problem-solving 
behaviors, even though thjs was a major goal of the program. If our ob- 
jective had been to document departures frdm an ideal set of rational be- 
haviors, we would indeed have found rich data sources within the RDU schools, 
many of which we have pre^snted in case- materials found elsewhere (Louis, 
Kell, Chabotar and Sieber, 1981). However, despite dep^tures- from the 
ideal entoodied in the RDU program,- on the whole we came away convinced that 
what occurred in the participating schools was different from more typical 
plahning and decision making behaviors. In addition, based on tha c-iteria 
of rationality that were discussed in Chapter 4, an unexpectedly large 
proportion of the schools were, in fact, acting in accord with program 
objectives. At each stage of the problem-solving process, we rated 50!S or 
more of the 90 schools in our intensive sample as having adhered to princi- 
pals of a sound decision-making process either to a "great extent" or a very 
great extent." As we have also noted at various points in the volume, this 
achievement did not occur in favorable or placid organizational contexts. In 
many cases, schools did not become involved in the program vpluntarily, and 
initial reactions were not favorable. In addition, schools were plagued by a 
variety of critical events which, in many circumstances, would be expected to 
interfere with reasoned decison making and i^lanning. 

The above conclusions should not be interpreted to mean that the RDU 
schools conformed to textbook versions of planning. Rather, based on our own 
previous experience in studying planned change in : hools and that of others, 
we were simply surprised at the degree to which considered, deliberate and 
reasonably well thought through choices were made and implemented. This 
raises two significant questions that can be addressed only partially. The 
first is why did the progr a m appear successful in stimulating s omething ap- 
proximating rat ionality of organizational behavior , and the second is why, 
1.^^ 1.^^000 fh» antivitiea. were only bits and piece s of the process 
being repeatedly applied in the schools? 
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Our observations, and much of the analysis presented cbove, suggest 
that the general strategy of providing an external support system composed 
of hunan and technological resources and relatively small amounts^of money 
may have directly affected the degree of rat lon'ality . Several factors 
account for this impact. First, the availability of resources and externally 
imposed expectations created conditions in many schools that allowed a 
temporary alteration of the normal isolation of teachers in classrooms. 
Although it required energy from teachers and administrators (as is e?videnced 
"by the level of in-kind res'ources), it also provided some partial compensa- 
tion, most typically by reducing "clessroom press" through providing release 
time for teachers on the teams. The role of outside helpers — particularly 
field agents--was in large measure to disrupt the equilibrium of the schools. 
Field agents were not only facilitators, they were also outsiders to whom 
many of the schools felt accountable. Because many of the projects provided 
formal ''milestones'' as guides against which schools could measure their 
progress through a problem-solving process, there was a structure that 
provided them with a rationale for the existence and duration of the "tempo- 
rary system" which most teachers found to be quite different "from the normal 
mode of decision making. Thus, v^ile the problem-solving activities were 
locally driven and not exteinally directed , they were conducted in a context 
where there was continuous external support and stimulation to perform in 
certain ways. 

The very reason why it worked — the existence of an external system 
which both supported and monitored decision making — also contributed to low 
levels of incorporation. Because of the dependence of the "temporary system" 
on external support structures and roles, learning a new set of skills (which 
many teachers reported they felt they had done) is not a sufficient cause of 
permanent change in organizational behavior. The new skills might allow a 
school to engage in a rational problem-solving process more efficiently on 
another jccasion, but we suspect the energy required to initiate and carry 
through this process may be far .iiore difficult to muster in the absence of 
external stimuli. 

We also question whether a problem-solving process such as that used 
in RDU can typically be incorporated into the more normal functioning of a 
sf-^^ol* The amount of energy that was required of teachers and administra- 
tors to make a serious attempt over tvtqi to three years to solve a locally 
identified problem produced, in many instances, a certain level of fatigue. 
It would be overstating the case to say that staff members had "burned out," 
but even in the most successful schools we sensed little enthusiasm among 
most staff members for starting up again in another area of the curriculum, 
while the teachers generally enjoyed the RDU problem-solving process, like 
many other time consuming activities, it was not something they wished to 
dive into again. Thus, the frequently voiced statements that "we'll do it 
again, but probably not as thoroughly" reflect school staff members' belief 
that participatory, schooUwide self-renewal activities can cut deeply into 
other important school and professional activities. 

There is an additional, important question that the data presented in 
previous chapters also address directly: how important is rationality for 
knowledge utilization and school improvement outcomes? The answer that can 
be given to this queston may vary depending on the definition of rationality 
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that IS used. On the one hand, the degree to whirh a rational decisio n- 
making process was followed appears to contribute less to the explanation of 
scchool outcomes than many who were involved in the program would have 
predicted. The analysis of simple correlations in Chapter 7 indicates that 
the coongruence of site behavior with textbook models of planning arxi prob- 
lem-solving IS associated with most knowledge utilization and school improve- 
ment outcomes. However, regression analysis and canonical correlations 
ouijgest that this factor is not that potent when compared with others that we 
measured. In the end, our conceptual schemas presented earlier in this 
chapter include the quality of the problem-solving process as- a contributor 
to only one school improvement outcome: organizational development. 

On the other hand, we may look at a broader definition of ration- 
ality, which includes more than the artifactas of planning behavior. The 
definition offered by Thompson (1967) focuses ont on planning activities per 
se, but on their outcomes: the reduction of uncertainty , particularly with 
respect to comre organizational activities. The major reductions in uncer- 
tainty associated with site involvement in the RDU program came from the 
access to high quality product materials, and to trainx'>g and technical 
assistance that would permit more effective implementation. Thus, the fact 
that a local team reached the decision to adopt higher quality products and 
to take advantage of the external human resources available from the pro- 
gram—both relatively typical behaviors in the schools tha*" we visited— was 
much more important than the protess by which these decisions were reached. 
Again, this suggests that the RDU program was successful in fostering ration- 
ality largely because of the resource structure that it provided, and not 
necessarily because it tried to impose major changes in the way in which 
decisions are made in schools. 

Before leaving the topic of rationality in schools, one final 
observation is necessary. We have argued that the resources provided by RDU, . 
namely information and t#iihnical assistance, wera important in achieving 
improved problem-solving practices and outcomes in schools. We must also 
point out, however, that externally developed resource and accountability 
structures, no matter how voluntarist ic, always have the potential of induc- 
ing their own non-rationaJ ity, German and McLaughlin (1975) have emphasized 
the problem of "oport unism," which involves inventing school problem^ to 
match the priorities of funding agencies and programs. Our own data suggest 
that opportunism may have significant costs t;o schools, at least from^ a 
resource conservation perspective, RDU revealed a few additional sources of 
institutional non-rationality that occurred as a consequence of the* avail- 
ability of external resources. First, in a few instances sch.^ls felt 
pfesured to adopt products that they were not enthusiastic about, because the 
one that had captured their fancies were not "validated" and were therefore 
not acceptable to the project management. In most instances, this resulted 
in multiple adoptions, with the preferred product receiving most of the 
site's energies, but with a formal adoption to meet project criteria also . 
being "implemented." A second source of nonrational behavior occurred 
because of the program's limitation to basic skills and career education 
problems, • Some schools that we visited (approximately 25%) would, under 
situations of greater flexibility, have identified other problems as more 
important had they been allowed to. Finally, in a number of 'schools that 
were involved, the "problem" in question was not one that seemed, either to 
them or to us, very severe. In some cases, the problem identification and 
solution selection process revealed that a perceived problem was not as 
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significant as was previously believed, or that the existing curriculum was 
38 good as most of the available "solutions." Nevertheless, most schools and 
projects found it difficult to terminate relationships, and schools that did 
not have very significant needs were carried through until the end of the 
program. Given the energy and resources necessary to carry through'an 
RDU-like process m schools, this may be assessed as a real opportunity cost 
for both projects and their clients. 

As a dissemination program, RDU vj^s, as we have observed, a narrowly 
focused demonstration. such, however^ it could not meet all, or even 

a large nunfcer, of pressing school needs. While program managers and policy 
makers may take heart at its success, they must also face its limitations. 
RDU was successful m part because it was limited ^o basic skills and career 
education and, ^mdefd, these are pressing issues for many schools. Our 
data even suggest that this focus on classroom-level and student achievementT 
problems may have helpipd to ^crease the impacts of the program. However, 
from the perspective of the user , a more permanent dissemination system, 
whether federally or state7f unded , * that operated m such a constrained 
fashion would have seriously limited utility. Schools have a broad set of 
needs, including management problems, staffing problems, • and curriculum' 
problems, that occur m areas where there is little R&D information, and 
few validated products. While our data support a dissemination focus in the 
sense that the characteristics of the products and the ways in which messages 
are sent are deeply important to producing desirable school outcomes, from a 
practical standpoint it is difficult to justify withholding less than perfect 
information, or less than optimal messages m the face of pressing problems 
that are perceived by users. Thus, this study should not, we belie.c, De 
used to support a Tecommendation that s RDU-like program be implemented m 
each state. Rather, the findings — particularly those elaborated in Chapter 9 
and m the other models — should be incorporated into the improvement of 
existing state and federal practices, which are often designed to serve a 
more comprehensive set of needs and problems, with more limiteo resources. 
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